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Distributed Channel Synthesis 

Paul Cuff 



Abstract 

Two familiar notions of correlation are rediscovered as the extreme operating points for distributed synthesis of a discrete 
memoryless channel, in which a stochastic channel output is generated based on a description of the channel input. Wyner's 
common information is the minimum description rate needed. However, when common randomness independent of the input is 
available, the necessary description rate reduces to Shannon's mutual information. This work characterizes the optimal trade-off 
between the amount of common randomness used and the required rate of description. We also include a number of related 
derivations, including the effect of limited local randomness, rate requirements for secrecy, applications to game theory, and new 
insights into common information duality. 

Our tool of choice for efficiently achieving distributed channel synthesis is the cloud mixing lemma (related to the concept 
of resolvability of a channel). This work elaborates and generalizes this tool. The direct proof (achievability) constructs a feasible 
joint distribution over all parts of the system using the cloud mixing lemma, from which the behavior of the encoder and decoder 
{2JQ[ is inferred, with no explicit reference to typicality or binning. 
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Index Terms 

Channel simulation, channel synthesis, cloud mixing, common information, random number generator, resolvability, reverse 
Shannon theorem, total variation. 

I. Introduction 

WHAT is the intrinsic connection between correlated random variables? How much interaction is necessary to create 
correlation? These are some of the curious inquiries that are illuminated by the distributed channel synthesis problem, 
^ i " introduced as follows: An observer (encoder) of a random i.i.d. source sequence Xi,X2, ... describes the sequence to a distant 
random number generator (decoder) that produces Yi,Y2, .... What is the minimum rate of description needed to achieve a joint 
distribution that is statistically indistinguishable (as measured by total variation) from the distribution induced by a memoryless 
channel? 

The nature of distributed channel synthesis is quite different than most problems in communication and source coding. 
T^lj- ■ The objective of mimicking a random process is significantly more stringent than, say, producing an output sequence that is 
Tj" I empirically coiTelated (jointly typical) with the input sequence. Here we require the resulting input-output pairs {Xt,Yt) to 
QQ ■ be nearly i.i.d. according to the joint distribution that a prescribed memoryless channel would imply. In previous work [IJ we 
I define two notions of coordination that distinguish this important point. This work is the "strong coordination" of ([T]. 

■ Remarkably, random bits available in common to both the encoder and decoder play a non-trivial role in channel synthesis. 
\ I [ Because of the unusual nature of the problem, common randomness can replace some (yet not all) of the communication, 

J> ■ providing a stochastic connection between the encoder and the decoder This stems from an important property of channel 
' synthesis — unpredictability. A properly synthesized channel will produce random outputs, free from perceivable patterns to all 

■ who do not see the communication and common randomness. 

d \ A particularly enticing use of distributed channel synthesis is in interactive adversarial settings. In the context of game 
theory, correlation constraints for actions among cooperating participants have been considered in the literature (e.g. |2| and 
||3|). We discuss the role of our channel synthesis results and the connection to secrecy systems in illll-BI In many repeated 
game settings, distributed channel synthesis provides the optimal means of communication. 

The distributed channel synthesis problem provides a fresh look at coiTelation. Many fruitful efforts have been made to 
quantify coiTelation between two random variables. Each quantity is justified by the operational questions that it answers. 
Covariance dictates the mean squared error in linear estimation. Shannon's mutual information is the descriptive savings for 
lossless compression due to side information and the additional growth rate of wealth in investing. Gacs and Korner's common 
information |4| is the number of common random bits that can be extracted from coiTelated random variables. It is less than 
mutual information. Wyner's common information ||5l is the number of common random bits needed to generate correlated 
random variables and is greater than mutual information. 

In the distributed channel synthesis problem, two quantities emerge as extreme points — Shannon's mutual information and 
Wyner's common information. Without common randomness, the required communication rate is Wyner's common information 
C{X; Y), consistent with Wyner's result in (5] concerning the minimum connection needed to generate correlated random 
variables. However, when enough common randomness is available, the communication requirement is reduced to the mutual 
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information I{X; Y) (consistent with f6l). These extremes are evident in the main result, Theorem lII.il and common information 
is discussed further in i HIIl 

Channel synthesis has emerged recently as a concept of interest in quantum and classical information theory. Bennett et. 
al. introduced a "reverse Shannon theorem" |6| (see also Q and JS)) which states that all channels of the same capacity are 
equally valuable. If one ignores encoding complexity and has unlimited common randomness available, then any memoryless 
channel can be used to synthesize any other channel of lower capacity. Referring to Shannon's theorem as the reduction of a 
noisy channel to a noise-free one, their reverse Shannon theorem uses a noise-free channel to synthesize a noisy one. This is 
precisely the problem considered in the present paper as well; however, we consider common randomness also to be a limited 
resource, yielding a trade-off between the use of communication and common randomness. 

Limited common randomness for distributed channel synthesis was considered by Winter in ||9l for a certain extremal 
operating point rather than the entire optimal trade-off. Winter's encoder was designed for a specific operating point and 
does not immediately generalize, but some of the proof methods are similar to oursjj Winter then connected these quantities 
(communication and common randomness for channel synthesis) to so-called "extrinsic" and "intrinsic" data in quantum 
measurements fl^. Further work on quantum measurements and exact channel synthesis can be found in ITSl . ifTOi l. and |15|. 

A major emphasis of this paper is the introduction of new proof techniques. Our construction of optimal codecs in [JV] is 
unusual in that we don't begin by stating the behavior of the encoder in an explicit, causal manner but instead construct a 
desired joint distribution and infer the encoder behavior from that. The crucial tool that we use is the "cloud mixing" lemma 
of i jlVI The development of this tool (related to "resolvability" in [16|) is a central part of this work, and we give a variety 
of derivatives and generalizations. Recently an alternative proof tool has been proposed in ITtI which uses a random binning 
construction to take the role of cloud mixing. 

We provide the main result and examples in SJII] followed by a number of extensions to the basic distributed channel synthesis 
problem in i jllll Other extensions to this problem are found recently in the literature (e.g. ifTSl . |fT9l , ll20l ). building on our 
introduction of the problem in ifTTl . ETI . and llj. 

II. Main Result 

A. Distributed Channel Synthesis 

Let {Xi} be a discrete i.i.d. random process, distributed according to Qx- The dashed box in Figure [T] represents a system 
that is designed to operate as if it were a memoryless channel defined by the conditional probability mass function Qy\x- 
However, the internal components of the box are constrained. Suppose the input and output of the channel are not co-located. 
An encoder which observes the channel input and a decoder which produces the channel output use communication and 
common randomness to synthesize the channel. The synthesis is successful if it cannot be distinguished through a statistical 
test from the memoryless channel that it is designed to mimic. This requirement will be clarified in §11-11 
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Fig. 1: Main Setup: We synthesize a memoryless channel across a distance by making use of communication and common 
randomness. The necessary and sufficient rates are given in Theorem lII.il 

The distributed channel synthesis problem asks what resources are needed to successfully accomplish this channel synthesis. 
The resources come in the form of a message J that is transmitted from the encoder to the decoder and common randomness 
K that is independent of the channel input. This work characterizes the required bit-rates of this communication and common 
randomness. 

We allow the system to operate on blocks of n inputs at a time, producing n channel outputs. This is the standard block 
encoding used in communication and compression. However, even within the block, the system mimics a memoryless channel, 
where the n outputs are conditionally independent given the n inputs. 

' Ultimately, the entire trade-off curve, the main result of the present paper, was solved concurrently and independently by Bennett, Devetak, Harrow, Shor, 
and Winter and can be found in |8|. A presentation given by Bennett 1 10| of unpublished work, unknown to us at the time, contains the complete trade-off 
and occurred at roughly the same time as our publication in llll . 
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Definition 1. The desired input-output distribution for a block-length n is the product distribution on the pair of sequences 
X" and specified by the probability mass function YV^=i Qx{xi)QY\x{yi\^i)- abbreviate this simply as 

YIQxQyix- (1) 

B. Encoder and Decoder 

For a block-length n, the encoder produces a description of the source sequence X" at rate R, represented by J G [2"^] ^ 
{1, ...,2"^}. A random variable K, uniformly distributed on [2"^"] and independent of X", represents the common random 
bits at rate Rq known at both the encoder and decoder. The decoder generates a channel output based only on J and K. 

The encoder and decoder are free to use randomization, and indeed they benefit from doing so. Accordingly, the encoder 
and decoder are described by conditional probability mass functions. 

Encoder: Fj^x^^k probability mass function). 
Decoder: Gy"\j^k probability mass function). 

Definition 2. An {R,R(),n) channel synthesis code /or input alphabet X and output alphabet y consists of an encoder 
Fj\x^\K and a decoder Gy^\j,k defined on the supports X" £ A"", F" £ 3^", J £ [2"-^], and K £ [2"^"]. 

C. Induced Distribution 

Aside from the common randomness K, the behavior of the encoder and the decoder are independent. Therefore, the 
combined behavior of the encoder and decoder results in a conditional distribution of the output and message J given by, 

Py^,.j\x^,k = F.]\X",K Gy"\.j,k- (2) 

Definition 3. The induced joint distribution of an {R,Ro,n) channel synthesis code is the joint distribution on the quadruple 
{X^ ,Y"' , J, K) resulting from applying the encoder and decoder to the channel input and common randomness. In other 
words, it is the probability mass function, 

PX",Y",J,K — Py'\.j\x^,k Px",k, (3) 

where, by construction, 

1 " 

Px",K(a;",fc) = ^l[Qx{x^). (4) 

1=1 

Definition 4. The induced input-output distribution is the marginal distribution of the induced joint distribution, assigning 
joint probabilities to only the input X" and the output as 

Px",y"(a;",?/") = ^ Px",y",j.K(x", y", j, fc). (5) 

D. Tolerance 

We say that the memoryless channel specified by Qy\x can be synthesized with rates (i?, i?o) for input distribution Qx 
if there exists an {R,Ro,n) channel synthesis code that induces the desired input-output distribution Y[QxQy\x- However, 
we allow for some tolerance. If we require exact synthesis then we forfeit some of the substantial benefit that compression 
provides. For example, consider distributed synthesis of the identity channel, which is equivalent to lossless compression. "Near 
lossless" compression of {Xi} can be achieved with a rate of R = H{X), but exact lossless compression (and exact synthesis 
of the identity channel) requires R = log \ X\. 

We might hope to efficiently achieve exact synthesis by using variable length communication, just as variable length codes 
such as Huffman codes allow for exact lossless compression while achieving efficient average description lengths. For distributed 
channel synthesis, a simple adaptation to block encoding would be to use an efficient channel synthesis code most of the time, 
which nearly synthesizes the channel, and with a small probability use an inefficient channel synthesis code (uncompressed 
communication) to implement the needed coiTection. This is similar to the approach taken in |6|, where exact synthesis is 
achieved using variable rate communication and an unlimited supply of common randomness. Also, exact synthesis is achieved 
in lfT4l using rejection sampling. But the steps for achieving efficient exact synthesis in our case of limited common randomness 
are not immediately obvious. 

Instead of exact synthesis, we allow some tolerance in the form of arbitrarily small total variation. In §11-11 we define and 
discuss total variation and why it is a meaningful metric of tolerance. But first let us state the main definition for this work. 
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Definition 5. A pair of rates (R, Rq) is achievable /or synthesizing a memoryless channel specified by Qy\x with input 
distribution Qx if there exists a sequence of (_R, Rq, n) channel synthesis codes, for n — 1,2, where 



lim 

n— >-oo 



= 0. (6) 

TV 



Px'^,Y'^-1[QxQy\x 
Let C be the closure of the set of achievable rate pairs (i?,i?o)-ll 

C ^ Closure {Achievable {R,Ro) for Qx,Qy\x} ■ 0) 

E. Main Result 

The main result of this paper characterizes the rate of communication and rate of common randomness needed to synthesize 
a discrete memoryless channel Qy\x with an i.i.d. input distribution Qx- This characterization is given in the definition of 
the following set S: 

{R, Ro) e 7^2 : 3Px.Y,u e D s.t. 

R > I{X;U), }, (8) 

R + Ro > I{X,Y;U). 

where 

r Px.Y,u : {X,Y)^QxQy\x, 
D = < X -U -Y form a Markov chain, } . (9) 

I \u\<\x\\y\ + i. 

Theorem II.l. For a discrete memoryless channel, 

C ^ S. (10) 



Rq 

A 



I ' ^ > R 

I{X-Y) C(X-Y) 

Fig. 2: Main Result: Theorem III. 1 1 gives a trade-off between the rate of communication and the rate of common randomness 
required to synthesize a discrete memoryless channel. At the extremes, with no common randomness the communication rate 
requirement is Wyner's common information C{X; Y), and the requirement reduces to the mutual information I{X; Y) when 
unlimited common randomness is available. 



Proof of this result is the subject of sections |IV] |V] and|Vl] Furthermore, from ||V]it is clear that the total variation tolerance 
decays exponentially fast with n. 

Two extreme points of the rate region S for distributed channel synthesis are manifested directly in the inequalities of ([8]l 
and illustrated in Figure |2] If i?o = 0, the second inequality dominates, and the minimum communication rate R is Wyner's 
common information ISJ, defined as 

CiX-Y) = min I{X,Y;U). (11) 

U : X-U-Y 

At the other extreme, we see that with unlimited common randomness, the communication requirement reduces to i? > I{X; Y). 
To see this, notice that the data processing inequality I{X; U) > I{X; Y) can be met with equality by selecting U = Y. 
This suggests that I{X, Y;Y) — R = H{Y) — I{X; Y) = H{Y\X) is a sufficient rate of common randomness to minimize 
the communication rate requirement. It turns out, as is shown in |[l], that sometimes a different choice of U minimizes the 

^We deal with the closure because our proof does not handle the boundary points. 



5 



communication requirement while requiring even less common randomness. The smallest amount of common randomness 
(after which additional common randomness does not benefit) is referred to as necessary conditional entropy in Ijj: 

H(Y]X) = min H(f(Y)\X). (12) 

^ ' f:X-f(Y)-Y ^•'^ ^1 ^ 

F. Example: Erasure Channel 

Let Qx be the binary symmetric distribution (i.e. Bernoulh-half), and consider the symmetric erasure channel Qy\x with 
erasure probability p depicted in Figure [3] 




Fig. 3: Erasure Channel. 



We now find the optimal rates {R. Rq) € .S by considering Markov distributions in D. Fortunately, the sparsity in the joint 
distribution Qx,y simplifies this optimization. 

For any Px,y.u G D, the Markov property constrains that for each value u in the support of U the conditional distribution 
Px,Y\u=u is a product distribution Px\u=uPy\u=u- These distributions must fall into three categories, shown in Figure ID 
because the events {X,Y) = (0, 1) and {X,Y) — (1,0) have zero probability. 



Category A 




e 







Category C 





Category B 1 

Fig. 4: Categories of Conditional Distributions: Conditional distributions Px,y\u= 
of three categories due to the sparsity of the channel. 



for the erasure channel must fit into one 



The three conditional distribution categories are as follows: 

• Category A: If Px\u=uPy\u=u PUts positive probability on y = 0, then X = with probability one. 

• Category B: The reverse occurs if Px\u=uPy\u=u puts positive probability onY = 1. 

• Category C: The only alternative to categories A and B is to put zero probability on F £ {0, 1}. 

We now show that it is sufficient to consider only distributions Px.y,u G D which have at most one value of u for each 
of the three categories in Figure |4] and are symmetric. Notice that this gives us a bound of |Z//| < 3 for this example, which 
is less than the nominal bound < | A"! |y| + 1 = 7. Also, the symmetry reveals a nice qualitative observation. The optimal 
construction of Px,y,u for synthesizing the symmetric binary erasure channel for symmetric inputs is a concatenation of two 
symmetric binary erasure channels, as depicted in Figure |5] 

Only one of each category: Consider a distribution Px,y,u where U has two values in its support that are associated with the 
same category of product distributions (Figure |4]i. Define W = f{U) as the label of the distribution category associated with 
U (i.e. values of U having the same product distribution category map to the same value of W). The data processing inequality 
says that I{X] W) < I{X; U) and I{X, Y\ W) < I{X, F; U). We simply need to verify that Px,y,w G D — in particulai-, that 
the Markov chain property X — W — Y holds. This follows because in each category either X or F is deterministic. 

Symmetry: The desired input-output distribution Qx,y is symmetric. Consider any candidate distribution Px.y,u G where 
U = {A, B, C} labels the category of the associated conditional product distribution Px,y\u- Define P^ y to be the flipped 
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distribution where X = 1 — X,Y — 1 — Y, and U is equal to U with A and B exchanged. Clearly Pj^ y (j 
D and produces the same point in S. It also has the same property that the value of U correctly labels the category of the 
product distribution. Now define the symmetric distribution P' x,y,u to be the average of Px,y,u and P^ y u- The convexity of 
mutual information with respect to conditional distributions gives Ip'{X;U) < Ip{X;U) and Ipi{X,Y;U) < Ip{X,Y;U). 
Furthermore, P'x.y.u S D because mixtures of distributions within a category always result in a product distribution. 




Fig. 5: Concatenated Erasure Channels: Any optimal point in the rate region 5 in (O for the symmetric erasure channel is 
achieved with a choice of U that constitutes a concatenation of two symmetric erasure channels. 



We are left with two parameters in selecting Px.y.u — the erasure probability pi of the first channel Pu\x and the erasure 
probability p2 of the second channel Py\u — and one constraint: (1— pi)(l — ^2) = We can now summarize the 

achievable rate region by substituting r = (1 — pi) > (1 — p) and evaluating the mutual information terms: 

r (i?,i?o) : 3re [1 -p,r*] s.t. ] 

5 = < R > r bits, \ , (13) 

[ R + Ro > hip) + r {I - h {^^)) bits, J 

where h{-) is the binary entropy function and r* = min{2(l — p), 1}. 

Ro 



1 

Hp) - 




Fig. 6: Erasure Channel Rate Regions: The boundaries of the achievable rate regions for synthesis of the p-erasure channel 
with symmetric inputs are shown for p = .05, .1, .15, .9, .95 from right to left. Transition points on the curve for p ~ .85 
are labeled, where q = 1 — p. 



Common Information: The common information Cq{X;Y) is found by evaluating the second inequality {I{X,Y;U)) at 
r*. For erasure probability p < 1/2 we see that r* — 1 (verified by calculus), which is equivalent to choosing U = X. The 
common information in this case is Cq{X;Y) ~ 1 bit. For erasure probability p > 1/2 we get r* = 2(1 — p), which is 
equivalent to choosing the channel Py\u to have 50% erasures. The common information in this case is Cq{X-,Y) = h{p). 
No calculus is needed to verify this choice of p*. Notice that H{X,Y\U = u) < 1 bit for each category in Figure |4] which 
is achieved with equality for p*. To summarize: 

Minimum Communication: The minimum communication rate required in the presence of enough common randomness is 
R > I{X;Y) = 1 — p bits, and the rate of common randomness needed to achieve this is Rq > H{Y\X) — h{p). This 
operating point corresponds to a simple synthesis strategy. The common randomness can be used to generate a list of erasure 
locations, and the encoder can then transmit the non-erased bits. 
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G. Example: Reverse Erasure Channel 

Let's now consider the reverse of this erasure channel example, by switching the input and output, as depicted in Figure [T] 
The channel input distribution Qx is symmetric on the set {0, e, 1} with probability p of erasure. The channel sorts the erasures 
randomly into O's and I's. 

X Y 




Fig. 7: Reverse Erasure Channel. 



The same derivation and parameterizations as above (erasure channel) hold for this example as well. The only modifications 
to S are the range of the optimal values of r and the necessary update to the first inequality: 

iR,Ro)en'^ : 3r e[r*,l] s.t. 

R > h{p)-rh{^) 
R + Ro > h{p)~rhC4^) 

where h{-) is the binary entropy function and r* = min{2(l — p), 1}. 



S 



(1 - p) bits, 
r bits, 



(15) 



Ro 




> R 



Fig. 8: Reverse Erasure Channel Rate Region: The boundaries of the achievable rate regions for synthesis of the reverse 
p-erasure channel with symmetric inputs are shown for p ~ .05, .1, .15, .9, .95 from right to left. 



H. Example: Scatter Channel 

Consider a channel Qy\x which acts on an input X G {1,2, ...,m} and produces an output uniformly at random from the 
same set excluding X. That is, 

QY\x{y\x) = ^-l(x^y). (16) 

771—1 

Now, apply to this channel a uniformly distributed input Qx and the result is a desired input-output distribution QxQy\x 
that is uniform over all pairs X j^Y. This distribution is repeatedly used as an example in |T|. 

We find the optimal rates {R, Rq) G 5 by considering Markov distributions in D. As in the previous example, the sparsity 
in the joint distribution Qx,y simplifies this optimization. 

For any Px,y,u G D, the Markov property constrains that for each value u in the support of U, the conditional distribution 
Px.Y\u=u is a product distribution Px\u=uPy\u=u- Each of these distributions must fall into one of (777 — 1) categories, 
distinguished by the quantity of their sparsity pattern. If Px\u=uPy\u=u puts positive probability on a points in X, for 
a = 1, 777 — 1, then it can put positive probability on no more than (777 — a) points in 3^, to avoid the possibility that X ~Y . 
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For each of the above categories, we have a trivial bound on conditional entropy: 

H{X\U = u) < logo, 
H{X,Y\U ^u) < loga + log(TO-a). 



(17) 
(18) 



Thus, {H{X\U),H{X, Y\U)) must be in the convex hull of the union of two dimensional regions defined by ( fTTI l and (fTsT l. On 
the other hand, the corner points of these regions can be achieved because of symmetry by selecting (for any a) the uniform 
distribution over the support and by choosing Pu to be uniformly distributed over all support structures of the category 
associated with a. Therefore, 



Convex Hull 



/ 



V 



(i?,i?o) G re 

R 

R + i?o 



> 
> 



3a £ [m — 1], a > m/2 s.t. 

log —> f , 

^ \ a{m — a) j ' 



(19) 



This is depicted in Figure |9] for m = 3, 5, and 7. 




Fig. 9: Scatter Channel Rate Region: The boundaries of the achievable rate regions for synthesis of the scatter channel with 
symmetric inputs are shown for \X\ = 3, 5, and 7 from thickest to thinnest. Larger alphabets have greater benefit from common 
randomness. 



The common information for this distribution was calculated in fill and can be obtained from the above region. Let \m 
take the value of m rounded up to the nearest even number. 

[ml 



Cq{X;Y) = 2 bits -log 



[ml - 1 



(20) 



Notice that this increases to 2 bits as the alphabet size m increases. This is the required communication rate when no common 
randomness is present. Contrast this with mutual information, which decreases to zero as m increases. 



lQiX;Y) = log 



m — 1 



lege. 



(21) 
(22) 



/. Total Variation 

We use total variation to measure the distance between the induced input-output distribution and the desired input-output 
distribution. Total variation between two distributions 11 and F on a set W is defined in the following way: 

l|n-r||Ty ^ max (n(5) - r(5)) (23) 

= max (T(S) - n(5)) . (24) 

sew 
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If W is countable and Tr{w) and 7(11;) represent the probability mass functions associated with 11 and T, then 

||7r-7||Ty = ^lk-7lli (25) 
= ^ X! 1'^^"') (26) 

tuGW 

= l|n-r||Tv. (27) 

Total variation has some important properties that make it an attractive measure of tolerance. First is statistical indistinguisha- 
bility. Consider a test that tries to detect a synthesized channel. The performance of any binary hypothesis test is characterized 
by two parameters: the probability of false positive (a); and the probability of false negative (/3). Let 11 be the null hypothesis 
(the channel is genuine) and F be the alternative hypothesis (the channel is synthetic). If the two hypotheses yield identical 
distributions, then accurate detection is impossible, and a + /3 — 1 (any value of (3 — 1 — a can be attained by adjusting the 
sensitivity of the test). In general, 

a + (3 > l-||n-F|lTy. (28) 

Therefore, if total variation is small, then reliable detection cannot be hoped for This is the objective of channel synthesis. 
Another important property of total variation comes in the form of a bound on expected values of bounded functions. 

\EnfiW)--Erf{W)\ < 2 f max l/Hl) ||n - r||j,^ . (29) 

This bound implies continuity of Ef{W), for bounded /, with respect to the distribution Pw, using total variation as the 
distance metric. If we know that the total variation is small between two distributions 11 and F and we care about the expected 
value with respect to 11 of a bounded function, then we can instead analyze the expected value with respect to F, which is 
guaranteed to be close. We use this technique when analyzing the payoff achieved in the game theoretic setting of illlll and in 
our related secrecy work in ll22l . Il23l . and ll24l . 

Other metrics of distance between distributions have been explored in related works, such as |25| and |5|. A metric 
of particular interest is normalized Kullback-Leibler divergence (ixL(n;F) (also referred to as K-L divergence or relative 
entropy). Wyner uses normalized K-L divergence as his tolerance metric for generating correlated random variables in |I5|. K-L 
divergence has some interesting implications of its own. For one thing, it is an information-theoretic measure closely related to 
other important quantities such as mutual information. More importantly, K-L divergence also has theoretical implications for 
hypothesis testing. When a binary hypothesis test is based on a sequence of independent observations, K-L divergence gives 
the optimal exponent with which one of the two probabilities of error (a or f3, depending on which direction K-L divergence 
is measured) approaches zero while the other error stays small (Chernoff-Stein lemma: ll26l . Il27l ). 

Total variation and K-L divergence bare implication on different regimes of hypothesis testing. Total variation can give 
guarantees about the indistinguishable regime, where hypothesis testing error is high, via ( |28] |. On the other hand, the Chernoff- 
Stein lemma (using K-L divergence) is an asymptotic result indicating how many samples must be obtained for accurate 
distinguishability. It doesn't give concrete guarantees about the indistinguishable regime. 

The difference between these two measures of fidelity (total variation and K-L divergence) for channel synthesis might be 
summarized in the following way. If you ask for a synthetic channel to mimic n outputs of a genuine memoryless channel but 
then intend to use it for much more than n outputs (by reuse of the synthetic channel), then eventually this artificial channel 
will reveal itself statistically. In this case, K-L divergence, normalized by the block length, gives insight as to the amount of 
time before it is easily detected. An arbitrarily small normalized K-L divergence means the detector can be forced to wait 
arbitrarily long to achieve a fixed high level of detection accuracy. On the other hand, if you indicate the number of intended 
uses of the channel during the design phase, then a high-fidelity synthetic channel, as measured by total variation, cannot be 
detected with any significant accuracy. Achievability in Definition |5] says that a person may choose any e > and n large 
enough and ask for a memoryless channel to be synthesized such that the total variation between the induced input-output 
distribution and the desired input-output distribution is less than e, no matter how small0 

It should be noted that total variation is neither a stronger nor a weaker tolerance metric than normalized K-L divergence. 
Unnormalized K-L divergence is a stronger metric than total variation — Pinsker's inequality reveals that total variation converges 
to zero as K-L divergence approaches zero. However, normalized K-L divergence is a weaker metric and forfeits this relationship. 
A reverse relationship holds for i.i.d. distributions. That is, if 11 ^ F (i.e. 11 is absolutely continuous with respect to F), and 
F is an i.i.d. discrete distribution of n variables, then 

dKLin-X) = o(^(^n + logp-^^j ||n-F||Tv), (30) 

'The fact that this can be achieved for any n large enough is significant. In contrast to rate distortion theory, for example, the tolerance criterion in 
Definition [5] is not normalized, so we cannot simply repeat high-fidelity channel synthesis codes designed for short blocks and expect them to meet the same 
tolerance for large blocks. Total variation accumulates. 
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as ||n — rllry goes to zero0 This means that an exponential decay of total variation, with respect to n (which can be achieved 
for distributed channel synthesis), produces an exponential decay in dxL(n;r) with the same exponent. 

The choice of tolerance metric does affect the achievable rate region in some important special cases of distributed channel 
synthesis. For example, when synthesizing the identity channel and using total variation as the tolerance metric, R > H{X) is 
the necessary and sufficient condition for achievability (ignoring the case of equality), according to Theorem III. II However, if 
normalized K-L divergence were used as the tolerance metric, then perfect lossless compression would be demanded, and the 
rate requirement would be i? > log \ This example also helps to highlight the different implications of the two metrics. 
Notice that if near-lossless compression is employed to synthesize the identity channel, the probability of error, which equals 
the total variation, can be made arbitrarily small for an arbitrarily large block-length. However, if a compression scheme for a 
fixed block-length is used repeatedly, then eventually an error case will occur where the input and output to the channel will 
be unequal. A detector can reliably distinguish, after observing the channel for much longer than the designed block-length. 
In fact, by always declaring the channel to be valid unless an error case occurs, the detector enjoys an arbitrarily small missed 
detection rate (for long enough observation) while achieving zero probability of false alarm, which corresponds with the infinite 
K-L divergence that the compression algorithm induces. 

For a careful comparison of inequalities involving total variation, K-L divergence, and normalized K-L divergence, see ||28]| . 



III. Extensions 

A. Broadcast Channel 

The main result of Theorem lII.il can be readily extended to a situation with multiple separate decoders together synthesizing 
a memoryless broadcast channel Qyi,...,y,„|x> each producing one of the channel output sequences after receiving a common 
transmission from the encoder as well as common randomness among all nodes. This is depicted in Figure [TO] The region of 
achievable rates for synthesis is given by 

r (i?, i?o) e 7^2 : 3Px,n,...,y,„,[/ e i?i3C s.t. ] 
Sbc = { R > I{X;U), ), (31) 

[ R + Ra > I{X,Yi,...,Ym;U). J 

where 

r Px,Y^......Y^,U ■■ iX,Y,,...,Yrn)r^QxQYu...,Y^\X, ] 

Dbc = \ (X, Yi, y,„) are conditionally mutually independent given t/, >. (32) 

I \K\<\x\\yi\...\y„.\ + i. J 



Broadcast Channel Qyi,...,y„,\x 



X 



Encoder 




Dec. 


1 




Dec. 


2 




Dec. 


m 



Y 



Y, 



Fig. 10: Broadcast Channel: This setting extends the main result to include the synthesis of a broadcast channel with separated 
decoders producing each channel output. The decoders each receive a common communication message as well as common 
randomness. 



This region can be proven using the same steps as the main result. 

'^This statement uses \29\ and [27, Theorem 17.3.3]. 

^Due to asymmetiy, this statement is only trae if K-L divergence is measured in one of the two du'ections — the direction that Wyner defined in (5). 
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B. Game Theory 

Consider a zero-sum repeated game between two teams. Team A consists of two players who on the tth iteration take actions 
Xt G X and Yt G y. The opponents on Team B take a combined action Zt G Z. The strategy sets X, y, and Z are finite. 
The payoff for Team A at each iteration is a time-invariant finite function TT{Xt, Yt, Zt): As a zero-sum game, the payoff for 
Team B is —TT{Xt,Yt, Zt). Each participant observes all actions from previous iterations, and each team wishes to maximize 
its time-averaged expected payoff. 

Let Team A play conservatively by assuming the best strategy for Team B. In the worst case (from the viewpoint of Team 
A), the expected payoff in the ith iteration is 

n* = min E 7r(Xf,yf,z(X*-\F*-i)). (33) 

Clearly ( l33T l could be maximized by finding an optimal mixed strategy P*x,y that maximizes min^g^: Ep.7r(X, Y, z) and 
choosing independent actions accordingly for each iteration. This would correspond to the minimax strategy. 

Communication Constraint: Now consider an additional constraint on Team A. The players on Team A have as their 
only means of coordinating their actions a secure channel of communication, limited to a rate of R bits per game iteration. 
Specifically, Player 1, who chooses the actions Xf, communicates at rate R to Player 2, who chooses YtU 

We say a rate R is achievable for payoff 11 if there exists a communication protocol that obeys a rate limit of R and produces 
average expected payoff no less than 11. That is, there exists a block length n and a random variable triple U) that 

has the conditional independence structure Xt — (C/, X*^^, F*^^) — Yt for all t and such that \U\ < 2"^ and 



n 

-En* 



n 

t 



> n. (34) 



Let Q be the closure of the set of achievable pairs {R, 11). 

We claim that optimality is obtained by producing i.i.d. actions with respect to a joint distribution. Define, 

r {R, n) e 7^2 : 3Px Y s.t. 

Go = I R > C{X;Y), 

[ n < mm.ez'E TTiX, Y,z). 

where C{X; Y) is the common information defined in (fTTT l. 

Lemma III.l (Optimal cooperative strategy). 

Q ~ Convex Hull{Qo). 

Comments: Notice that Lemma UlI. ll involves a convexification of Qc,. This means that it may be optimal to split time between 
two different efficient strategies — one that operates at a low communication rate and one that operates at a high communication 
rate — in order to satisfy the average constraint R (see Figure fTTTl. 

n 

A 

Combine two 
efficient strategies. 




R 



Fig. 11: Optimal Cooperative Strategies: Optimal efficient communication for a zero-sum repeated game utilizes one or two 
efficient strategies from Gq. 



Variants of this problem have been considered in ||29l . Il22l . ||23l . and ll24l . The difference in those works is that (the 
actions of Player 1 in this setting) are instead observed states of nature. Their distribution is not designed by Team A. The 
job of Player 1 is to compress and communicate the observed sequence efficiently to Player 2. If the communication occurs 



'Communication and common randomness play tlie same role in this setting. 
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over a public channel, with use of common randomness to conceal the communication, then the optimal solution is exactly 
characterized in |23| and is integrally related to the ability to synthesize a memory less channel. However, communication over 
a private channel, as in the present setting, is addressed in ||29J and still open. 
The proof of Lemma IIII.ll is in the appendix. 



C. Public Channel 

What if the communication used for distributed channel synthesis occurs over a public channel and we wish for the synthesis 
to be immune to statistical tests that utilize the public message J? We require X" and F" to pass as the input and output of 
a memoryless channel and J to appear unrelated to X" and y". That is, for rates (i?, i?o) to be achievable, there must exist 
a sequence of {R,Ro) channel synthesis codes such that the induced distribution Px",Y",j satisfies 



lim 



Y\X 



= 0. (35) 

TV 

This setting falls into the context of secrecy, related to |23 | and f22l|. Common randomness can be used as a secret key to 
encrypt the public communication. We find that this straightforward adaptation to distributed channel synthesis, where extra 
common randomness is used as a one-time-pad on the public communication, produces the optimal rate pairs [R, Rq). The 
closure of the set of achievable rate pairs is given by 

r (i?,i?o)e7^2 : 3Px.Y,u e D s.t. ] 
Spc = { R > I{X;U), ), (36) 

[ i?o > I{X,Y-U). J 

where D is defined in (|9|l. 

Surprisingly, the common randomness rate requirement i?o is greater than the communication rate requirement R for 
synthesizing a memoryless channel by use of a public communication channel. The common randomness requirement can 
be reduced to the common information Cq{X-, Y), and the communication rate requirement can be reduced to Iq{X; Y), but 
the two extremes cannot be achieved simultaneously in general. 

Proof is in the appendix. 



D. Limited Duration Fidelity 

Consider a relaxed objective for channel synthesis. Suppose the objective is to synthesize a memoryless channel with high 
enough fidelity that it would pass any statistical test with limited memory of length B. That is, for any e > we desire an 
encoding such that 



iXWY\X 



< 



TV 



(37) 



^Px,Y.u e P> s.t. 

IiX;U), 
bI{X,Y;U). 



bn. 



(38) 



where B may be much smaller than the encoding block n. 

The region of interest for a sharp rate requirement occurs when B grows linearly with the encoding block length: B 
In this case, the region of achievable rate pairs {R, Rq) contains the following region: 

r (i?,i?o)e7^2 : 
Slm = \ R > 

[ R + Ro > 

where D is defined in (|9|l. 

In particular this means that for finite memory B that does not grow with n, no common randomness is required and 
the communication rate must only exceed R > Iq{X;Y). Notice that the sum-rate bound in ( |38] | is dominated by the 
communication rate bound in (l38b when 72 is large enough that Iq{X; Y) > —Hq(Y). 

See the appendix for proof. 



E. Local Randomness 

The optimal encoder design for distributed channel synthesis, presented in [jV] calls for randomization at the encoder and 
decoder The randomization at the encoder is insignificant and perhaps even avoidable altogether It is easy to show, for example, 
that Hp{J\X", K) scales no more than linearly with n at a rate close to the arbitrarily small excess rate R — I{X\ U), where 
U is the auxiliary random variable in the region S of On the other hand, the private randomization required by the decoder 
is much larger The decoder of ijV]locally synthesizes a memoryless channel according to Py\u ™d applies the input k) 
from the codebook to this synthesized channel. 

Here we can quantify explicitly the amount of local randomness required by the decoder, similar to ll30l and ||3T1 . Let Rl 
be the rate of random bits L £ [2"^^ ] available to the decoder only, and define the decoder as a deterministic function 

G : JxICxC^y". (39) 



13 





J e [2"^] 


\ 1 




Fj\X"K 


G{J,K,L) 









Fig. 12: Local Randomness: In this extension to the main resuh, the decoder is deterministic but makes use of rate-hmited 
local randomness. 



This is depicted in Figure [121 

We now aim to characterize the set of rate triples {R, Rq, Rl) that can synthesize a memoryless channel Qy\x with input 
distribution Qx, and we conclude that the closure of this set is given by 

{R,Ro,Rl) e 7^3 : 3Px,Y.u e D s.t. 
R > I{X-U), 
R + Ro > I{X,Y:U), 



(40) 



Rr 



> 



H{Y\U). 



The total amount of randomness flowing into our synthetic channel (ignoring the minimally random encoder), when all 
inequalities in Slr of gOli are exercised with equality, is R^ + R^ ^ I{X, Y; U) - I{X; U) + H{Y\U) = H{Y\X). To our 
delight, distributed channel synthesis is efficient even compared to the local synthesis in ifSTl . 

This proof can be found in the appendix. 

F. Common Information Duality 

Two notions of common information were introduced at nearly the same time in the literature. One by Gacs and Korner H 
is defined as 



Cg-k{X:Y) ^ max H{f{X)). 

f : Hif{X)\Y)=0 



(41) 



The other common information by Wyner in [5] is stated in (fTTT i. For this discussion, we refer to Wyner's common information 

as CwiX;Y) 

Attention has been drawn in the literature to dual properties of these two quantities. For example, Cg-k{X: Y) < I{X; Y) 
and Cw{X;Y) > I{X;Y), where equality holds in both cases or in neither. Also, both can be viewed as extreme points 
for the common message rate in the Gray-Wyner network f32\. In this network, correlated sources are encoded jointly using 
three messages and decoded separately, each decoder receiving only two of the messages. The message received by both is the 
common message. If we imagine the three messages traveling down a cable to a midway point (Segment 1) and then splitting 
into separate cables to travel to the separate decoders (Segment 2), with the common message duplicated at the juncture, 
then a simple duality can be stated. When the sum rate of the first segment is efficient, the common message rate is at least 
Cw{X; Y). When the sum rate of the second segment is efficient, the common message rate is no more than Cg-k{X; Y). 

Here we emphasize another operational duality and how the present results enrich the operational symmetry. 



X' 



M 



M 



M 



M 



Y^' 



(a) Gacs-Komer (b) Wyner 

Fig. 13: Operational duality of common information. 



Figures |13a| and |13b| show two complementary settings. In the first, i.i.d. observations of correlated random variables are 
used by separate, independent nodes to generate the same random bits (with high probability). The rate with which random 
bits can be generated is Cg-k{X:Y). In the second, equal random bits are provided to two independent nodes which must 
produce a correlated i.i.d. sequence (with high fidelity as measured by total variation). The required rate of random bits is 
Cw{X\ Y). These results come directly from the original work in ||4] and ||5]. 

Now we add a communication link between the two encoders with a somewhat peculiar constraint. The communication 
is required to be independent of the output of the receiving node (nearly independent as measured by total variation). This 
alteration is depicted in figures I14al and I14bl 
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(a) Key Agreement 
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(b) Channel Synthesis 



Fig. 14: Operational duality with communication: The rates of randomness M for both situations relax to mutual information 
when communication (independent of the receiver output) is allowed. 



The setting of Figure [T4al has been studied for the purpose of secret key generation in ll33l . ||34| . ||35]| . Il36l . and ||37| . It is 
shown that with a high enough rate of communication, namely H{Y\X), the rate of extraction of random bits in agreement 
increases to the mutual information I{X;Y). 

The dual setting of Figure |14b| is solved by Theorem III. 11 With a high enough communication rate, the required rate of 
random bits reduces to the mutual information I{X;Y). Furthermore, a communication rate of H{Y\X) is sufficient in this 
setting as well (and necessary for most distributions). 

To see how this follows from Theorem lll.il consider the equivalent description of the distributed channel synthesis problem 
given in the beginning of fJV] That description applies exactly to this situation as well. 

For added curiosity, we can state the comer points {R, i?o) of the descriptions of the achievable rate region for the two 
settings in a way that suggests a deeper relationship. Both rate regions can be described as the union of simple regions, each 
defined by the choice of an auxiliary random variable. In the case of Figure I14al the simple regions are rectangles defined by 
an upper bound on R and a lower bound on Rq. In the case of Figure I14bl the simple regions are pentagons defined by a 
lower bound on R and a lower bound on the sum rate {R + Rq). In both cases, we now specify the corner points that define 
the regions. 

For the setting of Figure [T4al the corner points of the rate region are {R, R^) such that 

R = I{X;U), (42) 
Ro = I{Y;U\X), (43) 

for some U such that X — Y — U forms a Markov chain. See 1351 . 

For the setting of Figure [T4bl the corner points of the rate region are {R, Rq) such that 

R = I{X;U), (44) 

Ro = I{Y;U\X), (45) 

for some U such that X — U — Y forms a Markov chain. See Theorem lII.il 

IV. Cloud Mixing 
How many clouds must be in the sky before it is overcast? 

Consider a channel ^v\u. This is nothing more than a collection of distributions on V, indexed hy u gU. Given a particular 
input u, the output distribution $y|[/=ti can be thought of as a localized cloud. On the other hand, if a random variable U is 
the input to the channel, then the output distribution ^v{v) = J2u {u)^v\u i'^W) is ^ weighted mixture of clouds, weighted 
by 

Suppose we wish to construct the mixture distribution <i>y at the output — the same mixture distribution that would have 
occurred if U ^ were used as the input to the channel — by instead randomly selecting an input from a small codebook. 
Can we create the illusion of $y by mixing fewer clouds than the total space Ul How many clouds are needed to create 
overcast? The answer is that we can get away with much fewer clouds than \U\ if we are allowed some tolerance which 
vanishes as the problem size grows. H 

The main ideas of cloud mixing can be found in the literature. Our contribution in this section is a simple and intuitive 
proof with tighter exponential bounds and greater generality of the results. The concept was introduced by Wyner to prove his 
common information result ||5j Theorem 6.3]. There he uses normalized K-L divergence as his tolerance metric instead of total 
variation, but the resulting rate needed for memoryless channels is the same. The work of Han and Verdu's on output statistics 
of a channel [161 elaborates on this concept for general channels (using total variation). They use the cloud mixing concept to 
quantify what they call the "resolvability" of a channel. Recent work in IITtI and lf38l has developed alternative constructions 
and analysis tools for obtaining similar properties to what cloud mixing provides, motivated by our work in |11| and ll j. 

The lemmas provided in this section form the backbone of the proof of achievability in this work and in related work. 

^According to the World Meteorological Organization, overcast is reported when eight eighths ('oktas') of the sky are covered. 
*In order to produce <E>v exactly, we would need the whole space U, unless the transition matrix ^y\u has a null space. 
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Fig. 15: Cloud Mixing: A sparse collection of conditional distributions ^v\u=u can be mixed together to approximate a marginal 
distribution ^v, making the construction from individual components impossible to identify, like clouds in an overcast sky. 



A. Cloud Mixing - Memoryless Channel 

Our simplest and most crucial result of this section for this work involves a memoryless channel with memoryless input. 
Let $[/ be a distribution on hi that induces a distribution when applied to the channel ^v\u- For channel uses, the 
corresponding input-output joint distribution is then 

Input Channel 

(K=i'^u{ut)) {K=i'^v\u{vt\ut)) 



yielding the desired output distribution 



= W'^u,v{ut,vt), 
t=i 



(46) 
(47) 



t=i 



(48) 
(49) 



The following lemma states that we can nearly produce the desired output distribution using only a collection of 2"^^^'^^^)+'^^ 
channel inputs or 'clouds' G Z//", indexed by j, referred to as the codebook S'^"-'. In fact, we can even do so with equal 

weight given to each of the clouds, meaning a uniform distribution is applied to the codebook, as in Figure [16] 



J - Unif 




u"(J) 


f ^ 


yn 









Fig. 16; Cloud mixing - Memoryless channel: An i.i.d. output distribution $y is synthesized by randomly selecting a codeword 
from a codebook of m" sequences and passing it through a channel. Lemma II V. 1 1 gives a sufficient codebook construction. 



The criterion for nearly producing the desired output distribution is that the induced output distribution 

PV<V-) A V- (50) 

I I j=l 

has vanishing total variation from the desired output distribution as n increases. 

Lemma IV.l (Cloud mixing - Memoryless channel). Let B^'^'' be a random collection of 2"^ sequences in U"", each drawn 
independently from the i.i.d. distribution according to $[/. A memoryless channel specified by ^v\U induces an output 
distribution defined in ( 150b . This output distribution is random because B^"^^ is random. 

If R > I^{U;V) then the expected value of the total variation between the induced output distribution and the desired 
output distribution defined in ( |49l l vanishes with n. That is, 

R>I^{U\V) =^ lim E||Py,. -gy^ II = 0. (51) 
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Furthermore, for discrete channels, the expected total variation vanishes exponentially fast: 

^\\Pv^~QvA\tv < ^e--^": (52) 
where 7, which depends on the rate gap R — I^{U;V) and the information span s^{U ; V) (Definition^, will be specified in 

The proof of this lemma requires only two simple steps — first, separate out the atypical problem cases that don't happen 
often; and second, bound the variance of each mass point in the output distribution. To avoid redundancy, we combine all 
proofs for this section together in ijlV-CI 



B. Cloud Mixing - General Channel 

We now provide three general lemmas concerning cloud mixing, forsaking the memoryless channel assumption. These 
lemmas become useful tools for a variety of applications. Some will be utilized in this work. The first forms the backbone of 
all proofs in this section. 

Before stating our simplest and most straightforward bound for channels in general, we need to define information density. 

Definition 6. The information density for a joint distribution '^u.v a function on the space U x V specified by the 
log-likelihood ratio of the joint distribution to the product distribution: 

Notice that the expected value of the information density is the mutual information: 

E*i$„^(C/;F) = I^iU;V). (54) 

Lemma IV.2 (Cloud mixing - General channel). For any channel ^v\u o'^d input distribution ^u, we bound the expected 
total variation induced by a randomly compiled codebook. Let B be a random collection of inputs u{'m) GlA, m — 1, M, 
each drawn independently from $[/. For any t, 

^\\Pv-Qv\\tv < P^{^^u,viU;V)>T)+^^. (55) 

where Py is the output distribution induced by selecting a channel input uniformly at random from the codebook and Qv = 
is the desired output distribution 'Ylm&j ^U^v\U- 

From this result we derive Lemma II V. 1 1 and also a couple of useful generalizations. Let us next restate in Lemma IIV.3I a 
result from the work of Han and Verdii |16 Theorem 4]. This lemma identifies the sup-information rate (defined next) as the 
threshold for synthesizing an arbitrary sequence of channels, with no assumptions such as memorylessness0 They refer to this 
as the "resolvability" of a channel. 

Definition 7. The sup-information rate I^{U ; V) for a sequence of joint distributions y{n) of pairs of random variables 

{U^'^\V'^'^'^) is defined as 

U{U-V) = limsupi i<E. , , , , f[/("^t/(")V (56) 

where the lim sup is in probability with respect to $. That is, 

limsupWn = inf{T : P<i>(M/„ > t) ^ 0}. (57) 

Lemma IV.3 (Cloud mixing - Sequence of channels (Theorem 4 of lfT6l )). Given a sequence of channels and input distributions, 
specified by for n = 1,2,..., let .B*-"-* be a random collection 0/ 2"^ sequences in W*-"-*, each drawn 

independently from ^ij(n). Let the induced output distribution Py{n) represent the output distribution of the channel when 
the input is selected from the codebook uniformly at random, as in (15 Oi l for the memoryless case. This output distribution is 
random because S*-"^ is random. 

If R > I^{U;V) then the expected value of the total variation between the induced output distribution and the desired 
output distribution Qyi^) = which is X]«(")ew(") 'I'c/C") It/*") vanishes with n. That is, 

R>h{U-V) =^ lim E||Pi,(„, -gvwilj.v' = 0. (58) 



'Note that we only focus on one direction of the threshold — sufficiency of a codebook size greater than the sup-information rate. Han and Verdii prove it 
to be a necessary and sufficient requirement, after optimizing over input distributions. We only use this tool in achievability arguments, so we focus on the 
one direction. For memoryless channels, necessity can be shown through entropy arguments. 
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Our proof of Lemma H V. 3 1 differs from the proof in IfT^ . Their proof is built around a relationship between the log-likelihood 
ratio and total variation, encapsulated in Lemma 5 of fTSl. Despite some similarity between the proofs, they are fundamentally 
different and produce different bounds. We can follow the steps of their proof to arrive at an equivalent of Lemma |IV.2| and 
make a straightforward comparison to ( fSSl l. After making appropriate substitutions, 

I^\\Pv-Qv\\tv < P'i{''^u,v(U;V)>T)+2^ (59) 
+ P^{i^^^^iU;V)>logM) (60) 

where ^ = f{i<s>u vi^'i ^) ~ log Af) and / is defined as 



E2*, (61) 



- { -oo, I ; !!. ^^2) 

The first two terms in ( l59b are comparable to ( |55] |. especially for large M. The term in ( |60] | won't be relevant. But the term 
in ( |6T1 ) causes concern. Again borrowing the method used in lfT6l . this term can be broken into two pieces using an indicator 
function. The most favorable bound obtained this way, after substituting t' = + log Af for any rj > 3 and combining 
terms, is the following: 

^WPv^QvWtv < ^{§7) " P^{i^u,v{U;V)>t') (63) 

This inequality is dominated by ( |55] l in Lemma IIV.2I It will still affirm an exponential decay in the expected total variation 
for the discrete memoryless channel, as in Lemma H V. 1 1 but with a smaller exponent. 

Let us now identify the exponent 7 that we deferred to specify in Lemma II V. 1 1 and then give a final generalization. 

Now that we have defined the information density, we can define the information span as the difference between the sup 
and inf in probability of the information density. 

Definition 8. The information span s$([/; V) for a pair of random variables ([/, V) distributed according to ^u,v is 

s^{U;V) ^ snpi^„^iU;V) -Mi^^^iU;V). (65) 

The information span is simply the length of the smallest interval that contains the support of the information spectrum. 
Notice that for discrete distributions the information span is always finite and greater than the mutual information. 
The exponent for lemmas IIV.2I and IIV.4I is 




A better exponent (but less explicit to state) can be obtained using Sanov's theorem Ii39l rather than Hoeffding's inequality 

isi. 



Notice that for ^^Jlf,)^'' < 1, 



and for » 1, 



R^U{U;V y ' 
Si{U-V) 



(67) 



7 ~ -^{R-U{U;V)). (68) 
2 log e 

Now we modify Lemma IIV. 1 1 very slightly by combining one use of a different orthogonal channel to many uses of a 
memoryless channel. This small generalization is very handy for proofs of secrecy as in ll22l . Il23l . and ll24l . 

Lemma IV.4 (Cloud mixing - Concatenated channel). Given a channel specified by the conditional probability mass function 
n"=i ^vic/l"*^* and an input distribution '^w{w)JYt^^^u{ut), let B*^"^ be a random collection 0/ 2"^ 
sequences in W x W", each drawn independently from the input distribution. Let the induced output distribution Pzy^ 
represent the output distribution of the channel when the input is selected from the codebook uniformly at random, as in (15 Oi l 
for the memoryless case. This output distribution is random because is random. 
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If R > I^{U; V) and ^{W; Z) is finite with probability one, then the expected value of the total variation between the 
induced output distribution and the desired output distribution Qz,V" = ^z,V" vanishes with n. That is, 

J lim EllPz.y- -Qz.y-llTy = 0. (69) 

l^„^(W ] Z) < OO W.p. 1 JH-oo ' "IV 

Furthermore, for discrete channels, the expected total variation vanishes exponentially fast: 

E\\Pz,v- -Qz.yAlTV e 0(e-^") (70) 

as n ^ OO, where 7 is given in ( l66b . 

C. Cloud Mixing - Proofs 

We first prove Lemma IIV.2I from which the other resuhs of this section will all follow. 

Proof of Lemma \IV.2\ Recall that <!>[/ ^v\u represents an arbitrary source and channel, and the codebook size \B\ is 
M. The proof is stated in terms of discrete random variables, but each step holds for continuous random variables as well by 
replacing the appropriate sums with integrals. 

We start by categorizing (u, v) pairs into a typical set that will behave well under $, using the information density i<i>{U; V) 
as the criterion: 

At ^ {{u,v) : < r} . (71) 

The idea will be to separate out the contribution to Py coming from typical pairs (u, v) E At from the rest. Let us define 
two functions on V that sum to Py: 



^i(^) - ^J2'^y\ui^Pirn))^iiUim),v)eAT). (72) 

m—1 

1 ^ 

= ^ ^ <i>y|[/(z;|C/(m)) l(([/(m),«) ^ A), (73) 

m—1 

where 1 represents the indicator function, and [/(to) is the mth item in the codebook and is a capital letter to represent that 
the codebook is randomly generated. Under these definitions, 

Pv = P1+P2. (74) 

An important observation about the induced output distribution Py, which is random because the codebook is random, is 
that it is unbiased with respect to the desired output distribution Qv'- 

1 

EPy(«) = — ^ E $y|c;(«|C/(TO)) (75) 
m—1 

^=^' ^<^v\u{AU{l)) (76) 

= (77) 

- Qviv). (78) 

where (a) is an application of linearity of expectation to the definition of Py, and (b) is due to the symmetric nature of the 
codebook B. 

We separate the total variation E \\Pv — QvWtv '-^^ parts: 



^\\Pv~Qv\\tv = ^\\Pv-^Pv\\tv 

= \Y,^\Pv{v)~^Pv{v)\ 



2 

vev 



< i^E|Pi(«)-EPi(«)| 



2 

vev 



+ i^E|P2(t;)-EP2(^;)|, (79) 
vev 



where (a) is due to the triangle inequality. 
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The first sum in (|79]l is the interesting one to consider, so we save it for last. The second sum is easy to handle and is small 
as long as the typical set At is likely. Starting by again making use of the triangle inequality. 



2 

vev vev 



(a) 



i^E|P2(«)-EP2(«)| < ^EPaM (80) 

vev 

E E ( ]^ E '^v\uiv\Uim)) 1 mm), v) i Ar)] (81) 

1 ^' 

E M E E i'^vwivPim)) 1 ((C/(m), v) i Ar)) (82) 

v^V in—\ 

E {^v\uiv\U{l)) 1 ((C/(l), v) i Ar)) (83) 

vev 

1-P$(A) (85) 
= P$(z$„,,(C/;F) >t). (86) 

Equality (a) is again a consequence of the symmetry of the codebook. 

The remaining term in ( |79] l deals with only typical pairs. To bound this term we appeal to a variance bound with the help 
of Jensen's inequality: 



E|PiH-EPi(i;)| < A/E(i"i(«)-EPi(z;))' (87) 



- VVarPiH. (88) 
Since the input codebook B is a collection of inputs generated i.i.d., the variance of P\{v) breaks down nicely as 

VarPi(«) = Var ^ $yn,(z;|;7(TO)) l((t/(m),i;) e A)"] (89) 

\ m=l / 



M 



- ^ E Var($v|r/(i;|C/(m)) l((;7(m),«) e A)) (90) 

m— 1 

(1) l^Var{^v\u{v\U{l))l{{U{l),v)eAr)) (91) 

< j^E{^y^u{v\U{l))l{{U{l),v)eAr))^ (92) 



= j^J2'^u{u)i<S>v\uiv\u)liiu,v)eAr)y (93) 

ueu 

= J2 '^u{u)<^>\\uiv\u) (94) 

" E '^u,v{u,v)<i>viu{v\u) (95) 

< ^ 51 2-'i>u{u)<fv{v)<i>viu{v\u) (96) 
= — ^ $c/,y(u,w) (97) 

< (98) 

Equality (a) is due to the independence of the items in the codebook, (b) because of the symmetry, and (c) comes from the 
definition of Ar- 
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The conclusion with respect to the first term in (|79]l is 



IY.E\P,{v)-EP,{v-)\ < ^Y.J^<^2^iv) (99) 

vev vev 

1 

■ 

Proof of Lemma ITOl - With Lemma |IV.2I at our disposal, proving Lemma HV. 3 1 is just a matter of selecting t appropriately. 
If i? > V), then let i? > r > I<s,{U; V) and let r — nr in (l55T l. By the definition of sup-information rate, 

lim (z^ ([/("); > nr) = 0. (101) 

Also, since M ~ 2"^, the second term in (l55] t vanishes: 



1 / 2^^" 

■ 

Proof of Lemma \IV.1\ The first statement in ( |5T] i of Lemma |IV. 1 1 is a straightforward consequence of Lemma IIV.3I By 
the law of large numbers, 

/$([/; F) - I<i>{U;V). (103) 

For discrete channels we need to prove the more precise bound on total variation found in ( |52] ). Again, let us choose an r 
such that R > r > I<i,{X;V). However, this time we will choose it carefully, so that the exponential decay of both terms in 
(l55T l are equal. Let 



A = Wl + 16(loge) 2 /TT M' ^^^^> 

81oge sl{U;V) J 

r = /$([/; F) + A. (105) 
Now we apply Lemma |IV.2| with r = nr. First we bound the first term in ( |55l l. 

P* (i$„„,^„(C^";'^") > ^) - P* (^^^^u.viUk;Vk) > nrj (106) 

= P* |^^f^^^$,,,(C/fc;F,) -/$([/; F)>Aj (107) 



< exp ( ^2 ) (108) 



where 7 is defined in (l66T l. and (a) comes from Hoeffding's inequality ll40l . 
The second term in (l55T l is the following: 



1/2^ 1 /(r - 



2 V 2 lege 

Now, we sort out the numerator in the exponent using basic algebra: 

o2 ' ' 



(109) 



2V2- - 2^^n^l^J ^'''^ 

^xp r A^iAr^M^.V (ni) 



(112) 



X- {R- U{U;V)) = Jl + 16 1oge 1-8 lege 

Sloge \V si{U;V) s|([/; 



= -2(loge)7. (114) 

Therefore, 
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Proof of Lemma VlV.4\ The first part of Lemma |IV.4I in ( l69l l is again a straightforward consequence of Lemma HV. 3 1 The 
joint distribution of the input ([/", W) and output (F", Z) is the product distribution 

n 

iv,z(w,2)n*'^'^("«'"«)- ^^^^^ 

Therefore, the information density separates into a sum. The law of large numbers tells us that the sup-information rate is 
/$(/7; V). We safely ignore the contribution of Pw,z because it is finite with probability one. 

For discrete channels we follow the same approach as our proof of ( |52] | for Lemma II V. 1 1 Again, let us define A and r as 
in ( 11041 ) and ( 11051 ). However, this time define 

T = {n+l)r + I^{W-Z)-I^(U;V) (117) 

for use in Lemma IIV.2I 

To bound the first term in ( 1551 ) we use Hoeffding's inequality for independent but non-i.i.d. sequences ||40l (inequality (a) 
below). Notice that we have defined r in such a way that the expected values of the random variables in the sum are all 
accounted for: 

P*Kv.a",x,v4w^,c^";^,i^") >^) - v^(^^„,{w-z) + f2^i^^,,{Uk-,Vk)>^ (ii8) 

Recall the property for all positive x and y: 

^ > --4- (121) 

X + y X x^ 

Therefore, 

p. (...,,,„..„. («'.f/";^.i'")>.) < cxp(-g^ + ?^!|M2^) ,122, 



-2X^n \ ( 2\^sl{W;Z ) 



6XP I 2 fTT. 

fTT exp ,f;, ' (123) 



The second term in ( |55] ) is: 



1/2- 1 ^ (n + l)r + /..(W^; Z) ~ I^{U- V) - nR \ 

2 V 2^ - r^H j ^^'^^ 

1 ( \ + I^{W-Z) \ ( {r~R)n \ 
= 21oge ' ^'^'^ 



by the arguments found beginning at dl 101 ). 



D. Cloud Mixing - Variants 

In this section we provide a toolbox of results derived from or related to the cloud mixing lemmas presented in this section. 
Each of the results expresses a rate of random bits that is sufficient for producing nearly i.i.d. (in total variation) sequences 
of random variables with the assistance of a memoryless channel (in most cases). These results are presented in the form of 
self-explanatory figures and captions. Some readers may feel comfortable skipping this section or browsing the figures, as this 
generality is not needed for the proof of the main result. 

We begin with the basic cloud mixing lemma (Lemma II V. Il l illustrated in Figure [T7] The figure uses a double-lined arrow 
to depict a flow of random bits at rate R, which means that nR uniformly distributed random bits are available to the encoder 
in a block of length n. The encoding block in the diagram is labeled with the function notation /(•) to represent that it 
is deterministic. The block with curved corners represents a memoryless channel with behavior defined by the conditional 
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Fig. 17: (Lemma riV.lt : R > I{U ; V) is sufficient for producing an i.i.d. sequence. 



distribution ^v\u- For any input distribution $[/, the output V"^ will be nearly i.i.d. (using a random codebook construction) 
as long as i? > /([/; V). 

We focus only on one side of each question presented in this section — sufficient rates — because we anticipate applying these 
tools to proving achievability arguments for communication. However, in all of these cases, entropy arguments can be used in 
a trivial way to show necessity of these rates (up to a null-space in the channel transition matrix). This is done by counting 
the total contributions to entropy. As an example we will do this for the case of Figure [TtI 

Let M represent the random bits that enter the encoder. 

H{V'^) < H{M,U'^,V'') (128) 

= H{M,U'') + H{V''\U'") (129) 

= H{M) + H{V''\U'^) (130) 

< nR + H{V''\U"). (131) 

We know that H{V'"') must be close to nH{V) because $yn is close to the desired i.i.d. distribution in total variation (refer 
to Theorem 17.3.3 of [27]). The only tricky technical detail is to show that is close to nH{V\U). This can be 

done by first considering the marginal distribution of Ut for any t. Since the distribution of Vt is arbitrarily close to $v for 
all t (see Lemma IV. Il l, continuity and compactness implies that the distribution of Ut must be close to the preimage of 
through the channel. Finally, due to the memoryless property of the channel, the conditional entropy separates as 

n 

H{V''\U") = Y.H{Vt\Ut), (132) 
t=i 

and each term in the sum is close to H{V\U) by (|29] l for some Pu in the preimage of $y through the channel. 
Next, consider in Figure [18] a variant that does not include a channel. 

Corollary IV.5. R > H{U) is sufficient for producing an i.i.d. U" sequence in Figure\l8\ for any distribution ^jj- 



R 
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Fig. 18: ( Corollary \IV.5i : R > H{U) is sufficient for producing an i.i.d. U" sequence. 



Corollary I1V.5I is a special case of Lemma II V. 1 1 with the identity channel. 

Our next example is trivial but serves to highlight the notation of our figures. Notice in Figure [19] that the sequence 
produced by the encoder is shown protruding past the vertical dashed line. A successful rate and encoding in this setting must 
cause all signals that pass the dashed line to be correlated and i.i.d. 

Corollary IV.6. R > H{U) is sufficient for producing an i.i.d. multi-source (U", V") in Fisure [TP] for any input distribution 
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Fig. 19: ( Corollary \IV.6\ : R > H{U) is sufficient for producing an i.i.d. multi-source (C/", V""). 



Corollary I1V.6I follows immediately from Corollary I1V.5I because the desired conditional distribution of given [/" is 
exactly the channel that the setting provides (see Lemma [V2l i. 
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We now bring another i.i.d. source into the picture. In Figure |20l the sequence is i.i.d. according to the distribution 
and is an input to the channel. Again, the goal of the system is to produce y so that the two sequences and V" 
are correlated and i.i.d. according to ^w,v (close in total variation). This is indicated in the figure by both signals protruding 
past the dashed line. Another intuitive way to interpret this setting is that the system produces a memoryless channel from W 
to V. 

Corollary IV.7. R > I(U; V\W) = I(U; V, W) is sufficient for producing a sequence correlated with the sequence 
in Figure^O\(as an i.i.d. multi-source), for any product distribution ^u,w = ^u^w- 
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Fig. 20: (Corollary \IV.7i : R > I{U;V\W) = I{U;V,W) is sufficient for producing a memoryless channel from W to V, 
where U and W are independent. 



Corollarv lIV.7l is justified by interpreting VK" as a channel output along with V"". The channel is ^w,v\u = ^w^v\u,w so 
that W does not depend on U. Corollary IIV.7I is also related to the upcoming Figure |2T| with the additional assumption that 
the encoder doesn't use the sequence VF" even though it has access to it. 

In the next situation in Figure [21] the source W" is also available to the encoder We are able to extend the result in this 
case to a universal result for non-random sequences. 

Corollary IV.8. R > I{U;V\W) is sufficient for producing a V"^ sequence correlated with the (discrete) W" sequence in 
Figure^J](as in i.i.d. multi-source), for any joint distribution ^u,w- Furthermore, for any 5 > and n large enough, for all 
individual sequences W" satisfying R > ^ ^{^j V\W = Wt) + 6, the conditional distribution of V" will be arbitrarily 

close to the desired conditional distribution in total variation. 
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Fig. 21: ( Corollary \YV.8\ : R > I{U;V\W) is sufficient for producing a memoryless channel from W to V. This result also 
holds universally for an arbitrary sequence using variable rate codes with rate R — I{U; V\W) + 6 calculated according 
to the empirical distribution of W". 



Corollary IIV.8I follows from an interleaving argument. Since the encoder observes the sequence VF", the encoding can be 
designed for each individual sequence. Interpret as the interleaving of several different monotone signals, one for each 
value in the finite set W. Design a separate encoder for each of those signals using an effective rate of I{U: V\W = w) + e 
for each. Define the empirical distribution Tw" {w) to be the length of the signal with value w divided by the block length 
71. The total rate used is i? = E^ew ^VF- (w)(/(C/; V\W w) + e) = It{U; V\W) + e, where It{U; V\W) is defined with 
respect to the empirical distribution of W". 

The result of this encoding is \\V\ different interleaved output streams that are mutually independent. Each one closely 
approximates the desired conditional distribution to within a negligible uniform bound of total variation^ Therefore, the 
triangle inequality states that the total variation between the desired distribution and the distribution of the entire output 
sequence will also be negligible. 

This establishes the existence of an encoder which works universally for all such that R — It{U; V\W) (a function of 
the empirical distribution Tw^) is positive and bounded away from zero. Naturally, a variable rate universal encoder can be 
used to synthesize a channel from 14^ to y for an arbitrary input sequence W"". A fixed rate universal encoder would require 
a rate R > max^gw I{U ; V\W = w). 

'"a technical detail should not be overlooked. In order to get a uniform bound on total variation for all streams, each one must be long enough. The 
uniform bound e will apply to all streams such that nTyirn (ui) > A' for some that depends on e. For streams that are shorter, we may need to apply some 
extra random bits to the encoder that will vanish with respect to the block length. 
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For the case where ]¥"■ is an i.i.d. sequence, as long as i? > I{U;V\W), the law of large numbers provides that R — 
It(U ; « i? — I{U ; V\W) > with high probability for n large enough. Thus, the total variation of the joint distribution 

of (VF", V") will be close to ideal. In fact, this result could also be obtained as a special case of the upcoming Corollarv lIV.l II 
of Figure l24l where the channel output is {W, V). 

The extension in CoroUarv lIV.SI to a universal encoder for arbitrary W"" is powerful but also limited to this setting. For other 
settings, such as Figure|20l we lack proof of the existence of a single encoder that works well for all sequences. The expected 
total variation for random encoders is small for all W^" sequences, but even using a fixed rate R > max^gvv I{U; V\W — w) 
we are not able to show that one encoder exists that performs well for all The upcoming setting of Figure |23] cannot be 
achieved for arbitrary W^" except in a trivial special case. The setting in Figure |24] can be achieved for arbitrary M^" if and 
only if there exists a conditional distribution ^u\w that makes V independent of W (a property of the channel). In this case, 
for this ^u\w^ Figure l24l becomes a special case of Figure 1211 

The next situation in Figure |22] shows a situation where the source sequence M^" is only available to the encoder. 

Corollary IV.9. R > I(U ; is sufficient for producing a V" sequence to be correlated with the (discrete) W" sequence 

in Fisure \22\ (as in i.i.d. multi-source), for any joint distribution ^u,W- Furthermore, for any 5 > Q and n large enough, for all 
individual sequences W"' satisfying R > ^ -^(^^5 V\W = Wf) + 5, the conditional distribution of V" will be arbitrarily 

close to the desired conditional distribution in total variation. 



w 



R 



Fig. 22: (Corollary \IV.% : R > I{U; V\W) = I{U; V) - I{W; V) is sufficient for producing a memoryless channel from W 
to V. 



Corollary IIV.9I is a special case of Corollary IIV.8I of Figure IIV.8I with a channel that does not depend on W. It is also a 
special case of the upcoming Corollarv II V. 1 31 of Figure HV. 1 3 1 where the channel output is (W, V). 

The following settings involve a random source sequence W" but are not concerned with the joint distribution of (W", F") 
that is produced by the system. The only requirement is that the sequence be i.i.d. The source can be used as a 
resource to produce V". In the setting of Figure |23] is a channel input but not avaible to the decoder. 

Corollary IV.IO. R > I{U,V) is sufficient for producing an i.i.d. V"" sequence in Figure l23l for any product distribution 
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Fig. 23: i Corollary \YV.10\ : R > I{U ; V) is sufficient for producing an i.i.d. V^" sequence, where W and V are independent. 



The system in Figure |23] is no different from Figure [TtI where takes the role of channel noise; therefore. Lemma II V. 1 1 
states that R > /(J7; V) is sufficient. Notice that the distributions of U and W must be independent because VF" is not 
available to the encoder 

In the next setting in Figure |24l the random source sequence VF" is a channel input and available to the encoder. We only 
care to generate 1/" to be i.i.d. 

Corollary IV.ll. R > I{U,W;V) — H{W) is sufficient for producing an i.i.d. sequence in Figure \24\ for any joint 
distribution ^u,w- 

Corollary II V. 1 1 1 and the several remaining corollaries are derived from a generalization of Lemma IIV.2I We state the 
generalization here and give the proof in the appendix. 

Lemma IV.12 (Cloud mixing - General source and channel). For any source distribution ^w, codebook distribution ^{j\w, cind 
channel '^v\w,U> can bound the total variation induced by a randomly compiled codebook. Let B be a random collection 



25 



W 



R 



^ /(•) 



jjn 



v\u,w 



yn 



Fig. 24: f Corollarv UV.l li : R > I{U, W;V) — H{W) is sufficient for producing an i.i.d. V" sequence, for any joint distribution 



of inputs u{w) EU, w G W, each drawn independently from <&(7|n/. For any t, 

-^\\Pv - QvWtv < P$(W,.,.WC^;"^)-*$w(W^) >t) +^2^/2. (133) 

where the self- information i{W) — i{W; W), the distribution Py is the output distribution induced by applying the codebook, 
and Qv = the desired output distribution ^ ^w'^u\w^v\w,u- 
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Fig. 25: Lemma HV. 12l is a generalization of Lemma llV.21 Here a random variable W in the input to a deterministic encoder 
which produces an output U . A channel then acts on the pair (VF, U). Given any output distribution consistent with the source 
and channel. Lemma I1V.12I bounds the expected total variation between the desired output distribution and the distribution 
induced by a randomly constructed encoder. 



The next result also follows directly from Lemma HV. 121 Here in Figure |26] an i.i.d. source W"^^ is available to the encoder 
at a different rate than the output, specified by r. 

Corollary IV.13. R + rH{W) > I{U;V) is sufficient for producing an i.i.d. V" sequence in Fisure |26] for any pair of 
distributions <^w <^nd $(7. 
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Fig. 26: ( Corollary \IV.13i : R + rH{W) > I{U; V) is sufficient for producing an i.i.d. V^" sequence. 

Next we reduce to the setting in Figure |27] with no explicit rate of random bits. The encoder relies on the randomness of 
the source W'^"'. 

Corollary IV.14. rH{W) > I(U; V) is sufficient for producing an i.i.d. V" sequence in Figure\27\for any pair of distributions 
ond <i>(7. 















/(•) 


Jjn 


f \ 





















Fig. 27: ( Corollary \IV.14h ' rH{W) > I{U; V) is sufficient for producing an i.i.d. sequence. 



Finally, consider in Figure |28] a setting with no channel. 
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Corollary IV.15. rH{W) > H[U) is sufficient for producing an i.i.d. U" sequence in Fisure \28\ for any pair of distributions 
and 
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Fig. 28: ( Corollary \IV.15i : rH{W) > H{U) is sufficient for producing an i.i.d. [/" sequence. 



Notice that there is some flexibility in designing the codebook for corollaries |IV. 1 31 [IV. 141 and llV. 15] in the synchronous case 
where r = 1. The codebook can be constructed from a conditional distribution ^ij\w7 as in all cases where the source sequence 
is available to the encoder However, in these cases the sufficient conditions and the output distribution $y do not depend on 
the correlation between W and U . They only depend on the marginal distributions of each. Therefore, a satisfactory random 
codebook can be constructed according to any conditional distribution <i>c/|n/ that produces the same marginal distribution on 
U. 



Compilations: The versatility of this dictionary of cloud mixing results is articulated through a number of examples. First, we 
restate a result known in the literature (e.g. fiT\) depicted in Figure |29] that a memoryless channel can be (locally) synthesized 
using H{Y\X) bits per channel output produced. We arrive at this as a consequence of Corollarv lIV.81 



Channel Qy\x 



R 



X 



Encoder 



Y 



Fig. 29: Local Channel Synthesis: We synthesize a memoryless channel by making use of H{Y\X) bits of randomness, a 
consequence of Corollary IIV.8I 

Next, consider an elaborate setting with two correlated source sequences and Z". Only is available to the 
encoder, and we must produce as an i.i.d. multi-source (close in total variation). For this setting, we can combine 

Corollarv II V. 1 1 1 with the idea behind Corollarv lIV.7l That is, treat as an additional output from the channel which depends 
on PF" but not J7". Under that interpretation, Corollarv II V. 1 1 1 extends readily to this situation. 

Corollary IV.16. R > I(U, W\ V, Z) — H(U) is sufficient for producing a sequence correlated with the sequence in 
Figure^^(as an i.i.d. multi-source), for any Markov distribution ^u,W,z = ^w,z^u\W- 
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Fig. 30: Two-source Example: R > I{U, W; V, Z) — H{U) is sufficient for producing a memoryless channel from Z to V, as 
stated in Corollary IIV.16I 

In another setting, we can consider two sources of random bits feeding into two separate deterministic encoders. The output 
of the first encoder is fed into the second encoder, as in Figure 1311 



Corollary IV.17. The following three rate conditions are sufficient for producing an i.i.d. Z"" sequence in Figure 137] for any 
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joint distribution ^u,V- 

Ri > I{U;Z), (134) 

i?2 > I{U,V:Z)-H{U), (135) 

R1+R2 > I{U,V;Z). (136) 
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Fig. 31: Two-encoder Example: Sufficient rates for producing an i.i.d. sequence in this two encoder system are given in 
Corollary HyTT] 



To prove Corollarv IIV.17I we need only show how to achieve the two corner points. Notice that the point (i?i,i?2) = 
{I{U;Z),I{V\Z\U)) is achievable due to Lemma H V. 1 1 and Corollarv I1V.8I because the second encoder, g{-), synthesizes a 
memoryless channel from U to Z, universally for all J7" with the appropriate empirical distribution, which will occur with 
high probability from the random codebook. 

The corner point i?2) = {H{U),I{U, V\ Z) - H{U)) is achievable according to corollaries |IV5] and [mil where f7" 
is synthesized as an i.i.d. process. 



E. Cloud Mixing - Summary 

In this section we've developed the main tool that we use for the achievability proof in iJVl However, we also expanded this 
tool for its own sake. This section can serve as a reference for research in other communication problems, such as channel 
synthesis in different network settings or secrecy capacity. 

Lemma I1V.12I tucked away in i )lV-DI as actually the most general starting point from which all other results in this section 
can be derived. This result is an unusual relative of the basic cloud mixing or "resolvability" concept, not found elsewhere in 
the literature. 



V. Achievability 

A. Synopsis 

In this section we prove C D S. That is to say, for any rate pair {R, Rq) in the interior of the rate region specified by S, 
{R, Rq) is achievable for synthesizing the memoryless channel Qy\x with input distribution Qx- The definition of achievability 
in Definition |5] is about the existence of {R,Ro,n) channel synthesis codes. However, the same achievability criterion could 
be stated simply in terms of the existence of a joint distribution satisfying certain properties, taking the emphasis off of the 
causal relationship of how an encoder or decoder takes an input and returns an output. This approach can be followed for 
defining any of the familiar communication problems in information theory, but we find it particularly useful in this case. Let 
us demonstrate. 

Consider the induced joint distribution Px^X'^j.k defined in Definition[3] The rates {R, Rq) are achievable if for any e > 
there exists an N such that for all block lengths n > N there exists an induced joint distribution Px" .Y" .j,k satisfying the 
following properties: 

1) X" - (J, K) - y" form a Markov chain. 

2) X" and K are independent. 

3) X" is i.i.d. - Qx- 

4) \J\ = 2"-". 

5) \IC\=r'^<>.B 

6) \\Px^\Y" -l\QxQY\x\\rpy < e- 

This is simply an exhaustive list of all the constraints imposed by the construction of the induced joint distribution, except 6) 
which is the synthesis requirement. 

Our method of attack will be to construct a joint distribution Tx",Y",j.k that satisfies 1), 4), and 5) by construction. We 
will then use the cloud mixing lemma of ijlVI to show that 6) is satisfied while 2) and 3) are nearly satisfied. Fortunately, 
due to some basic properties of total variation, we can augment the joint distribution to exactly satisfy 2) and 3) while not 
destroying the other properties. 

"it is not necessary to require K to be uniformly distributed, but it doesn't hurt either. 
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The key idea for developing this proof was to relax some of the strict requirements (properties 2) and 3)), knowing that this 
relaxation could be corrected at the end. By doing so, we reveal a large degree of symmetry in the problem statement. Rather 
than design the joint distribution from left to right (referring to the Markov chain in property 1)), we design from the middle 
outward. 

The consequence of this technique is that we design the encoder in reverse. Unfortunately, this makes it less natural to 
interpret. Its behavior may in fact be similar to other familiar encoders based on joint typicality, but our proof does not reveal 
this. Recent works such as IfTTl . |f38l . ||T9l . and ||20l are to that point and provide alternative methods of proof, building on 
this work. 



B. Construction 

Begin by finding Qx,y.u S D defined in ^ such that R > Iq{X; U) and R + Rq > Iq{X,Y; U). Our reuse of the 
label Q is intentional. By the definition of D, the marginal distribution of Qx.y.u must coincide with the desired input-output 
distribution specified by Qx Qy\x- 

Using the standard practice of random codebook construction to prove the existence of a good codebook, generate a codebook 
yB^"-' of u" sequences indexed by j £ [2"^] and k £ [2"''^''] independently according to J\t=i Qu{ut)- Now we construct 
iteration one of the joint distribution as follows. Define Tx".Y".,/,i<- such that J and K are uniformly distributed over their 
supports, and X" and Y" are the result of the codeword u"( J, K) passed through the memoryless channel defined by Qx,y\u- 
Notice that the channel Qx,y\u separates as Qx\u Qy\u because of the Markov chain property of all distributions in D. So 
we have, 

Tx",F",j,K(x",2/",j,fc) = (\{Qx\u{xt\ut{jM] [\{QY\u{Vt\ut{jM] ■ (137) 



\t=i 



At this point we know that T x^ ,Y" .,j,k satisfies properties 1), 4), and 5) by construction. Our next step is to construct 
Px" ,Y" ,j.K from Tx^,Y",j.K in a way that satisfies properties 2) and 3): 

PX^,Y^J.K = ^{l[Qx)rY'^,J\X^,K. (138) 

The conditional distribution Tyn j|X",/s- is derived from Tx'^,Y",j,k and well defined for all values of {X^, K) with positive 
probability. For others values we can simply assign the uniform distribution over 3^" x J . 

Notice that Px" ,Y" ,j.k satisfies property 1) because Py"\X",,j,k — '^Y"\X".j,k — ^Y"\j,k- It satisfies 2), 3), 4), and 5) 
as well by construction. Only property 6) is left to be verified. 



C. Synthesis Analysis 

Recall that T x'^.Y",.J.k and Px",Y",j.k are random because the codebook S'^"^ is random. We now call on the cloud 
mixing lemma (Lemma ll V. 1 b twice. First, we have a very straightforward conclusion. Since R + Rq > Iq{X,Y; U), 



lim E 

n—¥oo 



T 



X^,Y" 



WqxQ 



iWY\X 



0. 



TV 



(139) 



The second use of Lemma HV. 1 1 is a little more subtle. Notice that for any fixed k, the collection is a collection 

of 2"^ randomly generated codewords. If we consider only the memoryless channel specified by Qx\u with channel output 
X", then R > Iq{U;X) satisfies the condition of the lemma. Therefore, for any k, 



E 



- X"\K=k 



!X 







TV 



(140) 



as n — oo. The expression on the left-hand side is constant for all values of k because of the symmetric nature of an i.i.d. 
codebook. By the definition of total variation. 



E 



"^X",K — 



TV 



1 " 



-A; 



V 1 

OnB.0 O 



Tx^\Kix''\k)-l[Qxixt) 



4i: 



Tx..|if(a;"|l)-nfc(a:t) 



\TV 



(141) 
(142) 

(143) 

(144) 
(145) 
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Thus, Tx^.Y'^.j,K satisfies property 6) by (|139l l. and it nearly satisfies properties 2) and 3) by ( I141l i. 



Lemma V.l (Total Variation of Marginal). Total variation cannot be larger between marginal distributions than between 
encompassing joint distributions. That is. 



in 



w 



wWtv 



the left-hand side of ( 1146b is a maximization over a 



Proof: Referring to the definition of total variation given in 
smaller set than the right-hand side. ■ 

Lemma V.2 (Total Variation with Common Channel). When two random variables are passed through the same channel, 
the total variation between the resulting input-output joint distribution is the same as the total variation between the input 
distributions. That is. 



w 



^w^z\w\ 



TV 



Proof: Referring to the equivalent definition for total variation given in 
left-hand side of (11471 ) factors out of the absolute value and sums to one. 

We continue with the final steps of the analysis of Px",y" using the triangle inequality: 



\Iiw-Tw\\Tv (147) 
the non-negative Hziw isrm from the 



Px^ 



XWY\X 



TV 



< 



(a) 
< 



x^.^Wtv 

Y\_QxQy\x 



Px",Y",J,K — ^ X^,Y^,J,K 



'X",K — ^X".K\\xv 

+ \\rx^,Y- - Y[QxQ 

X I — ^X'^.K 



Y\X 



+ \\^X-.Y^ -1[QxQy\X 

Both terms vanish as n cxd because of ( I141l i and ( |139l l. Inequality (a) comes from Lemma IV. 11 and (b) comes from 
Lemma IV.2I 

Therefore, for n large enough, there exists a distribution satisfying all properties for achievability. ■ 
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TV 


(149) 
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(151) 
(152) 


TV 


(153) 


TV 


(154) 


TV ' 


(155) 



D. Comments 

We now summarize the behavior of the optimal encoder and decoder constructed in this section: 

Encoder: The encoder refers to the codebook B'^"-' of w" sequences indexed by j and k and considers only the subset where 
k is equal to the common randomness observed. In other words, the common randomness selects one sub-codebook for use. 
The encoder then considers each u" sequence in the sub-codebook and selects one randomly with probability proportional 
to the likelihood associated with the sequence u" passed through the memoryless channel Qx\u ^"d the observed source 
sequence X". 

It may happen that every codeword has a positive probability of being selected by the encoder; however, most of the 
probability will be concentrated on those codewords that are jointly typical with X". Still, there are many jointly typical 
sequences to choose from randomly. An interesting endeavor would be to design a deterministic encoder that successfully 
operates throughout the same rate region, if possible. 

Decoder: The decoder identifies a codeword w" given by the codebook Z?^"-', the common randomness K, and the message 
J. He then locally synthesizes a memoryless channel according to Qy\u to produce F" from m". 

In the decoder's case, a specific amount of randomization (H{Y\U) bits per channel use) is fundamental to the design and 
unavoidable according to ijlll-EI 



VI. Converse 



In this section we prove C C S. That is, any achievable rate pair {R, Rq) for synthesizing the memoryless channel Qy\x 
with input distribution Qx must fall in 



'^The set <S is a closed set. 
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A. Cardinality Bound 

The cardinality bound on the auxiliary random variable U in the definition of _D in (|9l) not only makes the region computable 
but is an essential step in the converse, as will be apparent in Section IVI-DI 

Lemma VI.l (Cardinality Bound). For any discrete random variables {X, Y, W) ^ ^x.y,w forming a Markov chain X — 
W ^Y, there exists a distribution T x,y,u forming a Markov chain X — U — Y such that 



M < 1-^113^1 + 1, (156) 

r^Y = nx,y, (157) 

Ir{X;U) = In{X;W), (158) 

Ir{X,Y;U) = In{X,Y;W). (159) 



Proof: Consider the set of points A G TZ\'^\\y\+^ such that the first l-^HJ^I coordinates represent the mass values of any 
product (independent) distribution PxPy and the last two coordinates are Hp{X) and Hp{X^Y). This is a connected set 
because each coordinate is a continuous function on the connected set of all product distributions. 

Let I^x.Y,w be the distribution of the Markov chain X — W — Y m question. Consider the point tt S 7?.I'*II-^I+^ where 
Tlx,Y specifies the first |<^||3^| coordinates and Hii{X\W) and Hyi{X,Y\W) the last two. Notice that tt is in the convex hull 
of A. It is a convex combination of points, each represented by a particular value of w, with convex weight equal to I{w{w)- 
The constituent product distributions are the distributions of (X, Y) conditioned on that value of W, and the entropies of the 
conditional distributions are averaged by the same weighting to give conditional entropies. 

The connected set A is actually contained in a (| A"! |3^| + l)-dimensional subspace of 'R}'^\\y\+'^ because of the linear constraint 
that a probability mass function sum to one. Thus, the Caratheodory theorem for connected se (see original publications: 
Bn for compact sets, [421 for general sets, [43 1 for connected sets; application to cardinality bounds of auxiliary variables: 
BU, ll45ll . Il46ll ) states that tt is a convex combination of (| A"! + 1) points in A. Associate each point with a value u. We use 
these points to construct the distribution Tx.y,u- The convex weight of the points becomes Yu{u), and the associated product 
distributions are the conditional distributions Tx^y\u=u^ yielding the desired Markov chain property X — U — Y . Notice that 
the joint distribution of (X, Y) and the conditional entropies are preserved by the construction of tt. ■ 



B. Entropy bounds 

A few preliminary bounds are needed to show that sequences that are nearly i.i.d. in total variation will have information 
properties close to their i.i.d. counterparts. 

Lemma VI.2 (Total Variation of Random Sample). The total variation between the distributions of two random sequences 
is an upper bound on the total variation between the distributions of the variables in the sequence at a random time index 
(independent of the sequence). 

Let T g {1, n} be a random time index distributed according to TIt- Also let Ilwn and be distributions independent 
ofT, so that Hvi/n^T = Hvi^nllT and Fn/n ^ = Tw^^t- Then, 

WWwt -^WtWtv - llnw" - Th/" IItv ■ (160) 

Proof: Notice that the channel niy^nyn(a|?i;") — X]t=i 11^(^)1 (a — Wt) defines the process that takes VF" and selects 
a random time index according to Ht- This Lemma simply requires that output distributions from a common channel are as 
close as input distributions in total variation — a consequence of Lemma IV. II and Lemma IV. 21 ■ 
Now we build on the fact that for finite alphabets we can upper-bound the difference in entropy in terms of total variation 
||27. Theorem 17.3.3]. 

Lemma VI.3 (Entropy and Timing Information of Nearly i.i.d. Sequences). For any discrete random sequence ~ Hw" 
where Wt G W for all t £ {1^ n}, if there exists a distribution Tw on the alphabet W such that 



then 



n 

and for any random variable T G {1, n} independent of W 



niy. -ffrw < e < 1/4, (161) 

1 " / 1\ 

-Y,In{Wt;W'-^) < 4e(log|W|+log-j , (162) 

t=i ^ 



MWt; T) < 4e ( log \W\ + log i ) . (163) 



'^This theorem is often refeired to as the Caratheodoiy-Feiichel-Eggleston theorem. 
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Proof: We start by applying Lemma IVI.2I for the arbitrary random time index T referred to in the Lemma as well as for 
each individual deterministic time index (each a special case of IIt). For each t e {1, n}, \\Iiwt ~ ^w\\tv < £• Then by 
Theorem 17.3.3 of 1271, 

iHniW) - Hr(W'')\ < 2elog fM^V (164) 



iHniWr) - Hr{W)\ < 2elog I ^ ) , (165) 



\Hn{Wt) - Hr{W)\ < 2elog ( ^ ) (166) 



for all t e {1, ...,n}. 

As with any i.i.d. distribution, iJr(W^") = Hr{Wt)- Therefore, the triangle inequality yields. 



^J2lniWt;W'-') = ^ (Jj^HniWt)^ - HniWn^ (167) 

< - \Hu{W"-) - Hr(W)\ (168) 
n 

1 " 

+ -y^\Hn{Wt) - Hr{W)\ (169) 
t=i 

(a) 2 /iwr\ /|W|\ 

< -e log f ^ j + 2e log U^] (170) 

n + 1 1 

= 4elog|yy| + 2elog- (171) 

n e 

< 4e (^log|W| +logi^ , (172) 

where (a) refers to ( I164l i and ( II66I 1. 

Furthermore, denoting the distribution of T as Ht, notice that 

n 

HniWrlT) = Y.nT{t)Hn{Wt) (173) 
t=i 

because of the independence of T and PF". Therefore, 



\Hn{WT\T) - Hr{W)\ 



J2^T{t) {HniWt) - Hr{W)) 



t=i 



(174) 



< Y.^T{t)\Hn{Wt)-Hr{W)\ (175) 

< ^HtW 2elog(^^j (176) 
= 2elog('^V (177) 



where (a) refers to ( |166t . Combining this with il65i gives 

MWt; T) < 4e (^log |>V| + log , (178) 
by way of the triangle inequality. ■ 

C. Epsilon Rate Region 

Now we use information theoretic inequalities and lemmas IVI.3I and IVI.ll to nearly complete the proof. We define a region 
iSe for e > that gracefully expands the region iS of the main result. Then we show that an achievable rate pair {R, Rq) is in 

Let the epsilon rate region be defined as 

(i?, i?o) e Ti^ ■■ ^Pxy.u e s.t. 

^ { R > IiX;U), }, (179) 

R + Ra > I{X,Y;U)-2g{e). 
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and 



where 

Px.Y,u ■ \\Px,Y - QxQY\x\\rpy < e, 
= { X ~U ~Y form a Markov chain, ) , (180) 

1^1 < 1-^11^1 + 1- 

5(e) = 4e (^log|A'|+ log 13^1+ log . (181) 

Lemma VI.4 (Epsilon Rate Region). If the rate pair (R, Rq) is achievable for channel Qy\x ond source Qx, then 

iR,Ro) e 5, Ve>0. (182) 

Proof: Since shrinks with e, let us only consider e < 1/2. Let {R,Ro) be achievable. Then there exists an (i?, i?o,n) 
channel synthesis code such that 



Px'^.y^ -IIQxQy\x 



< e. (183) 

TV 

Let the random variable T be uniformly distributed over the set {1, n} and independent of the induced joint distribution 
Px" ,Y" ,j,K- The variable T will serve as a random time index. The variable Xt is independent of T because X" is an i.i.d. 
source sequence (see |[T|, Property 1). However, Yt need not be independent of T. 

We lower bound R by, 

nR > Hp{J) (184) 

> Hp{J\K) (185) 

> Ip(X'']J\K) (186) 

Ip{X";J,K) (187) 

n 

= Y,IpiXuJ,K\X*-^) (188) 



n 

[b) 



J2lpiXuJ,K,X'-') (189) 



t=i 



> ^/p(Xt;J,i^) (190) 

= nIp{XT;J,K\T) (191) 

nIp{XT;J,K,T), (192) 

where (a) comes from the problem statement and (b) and (c) are due to the i.i.d. nature of X". 
Similarly, we lower bound the sum rate by, 

niR + Ro) > Hp{J,K) (193) 

> /p(X",y"; J,ii:) (194) 



= Y,Ip{Xt,YuJ,K\X'-\Y'-^) (195) 
t=i 

n n 

= Y.Wt,YuJ,K,X'-\Y'-^)^J2Wt,yuX'-\Y'-^) (196) 

(a) " 

> ^Jp(Xt,Ft; J,i^,X*-i,y*-i)-n5(e) (197) 

71 

> Y.Wt,Yt;J,K)~ng{e) (198) 
t=i 

= nIp{XT,YT:J,K\T)-ng{e) (199) 
= nIp{XT, Yt; J, K, T) - nIp{XT, Yt; T) - ng{e) (200) 

(6) 

> nIp{XT,YT;J,K,T)~2ng{e), (201) 



where (a) and (b) are both consequences of Lemma [Vl. 31 and g(e) is defined in ( 11811 ). 
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Notice the Markov chain given by Xt — {J,K,T) — Yt- This comes about because the entire sequences X" and are 
conditionally independent given J and K, according to the problem statement, so in particular conditional independence holds 
for Xt and Yr for any specific value of T = t. Therefore, by Lemma IVl. 11 we can find a Tx,y,u such that \U\ < \X\\y\ + 1 
and 



Ir{X;U) 
Ir{X,Y;U) 



< 1-^113^1 + 1, 

= Ip{Xt;J,K,T) 

= Ip{Xt,Yt;J,K,T). 



(202) 
(203) 
(204) 
(205) 



We see from (1184b and ( |193l l that rx,Y.u satisfies the inequalities in (|179l l. What remains is to verify that Tx.y,u G D. 
This is indeed confirmed by applying Lemma IVI.2I 



\^X,Y — QxQy\x\ 



TV 



TV 



< 
< 



Pxt.Yt ~ QxQy\x 
Px'^.Y'^ -YIQxQyix 



TV 



(206) 
(207) 
(208) 



D. Continuity of Se at Zero 

The final step in the proof is to show that the intersection of all Sc with e > is equal to S, a closed set. This may seem 
like unnecessary detail. It may seem obvious because of how was deliberately designed, namely Sq — S, and the non-strict 
inequalities in the definition of S seem to make it a closed set. 

There are a few subtle points to consider. Yes, S is closed, but this assertion relies on the cardinality bound of U. Also, 
notice that allows not only a relaxation in the sum rate but also a relaxation in the set of distributions D^. We need to 
make sure that a distribution near the desired input-output distribution does not have a much larger achievable rate region, 
as bounded by S^- Notice that in other work, such as |1|, this distinction is avoided by defining the achievable region as the 
closure of the set of achievable rates and distributions. In the present work, we define the achievable region as the closure of 
the set of rates for a given distribution — a more precise characterization of the achievable set — which requires this additional 
precision in the proof. 

Lemma VI.5 (Continuity of Se at Zero). The epsilon rate regions Se decrease to the closed set S as e decreases to zero: 

S. (209) 



e>0 

Proof: One direction of equality is trivial because Se shrinks as e shrinks and Sq = S: 

f]Se D S. (210) 

Notice that lime_j.o g{e) — 0. 

Let's first take care of the easy part. Define iS^ to remove the relaxation in the sum rate: 

{R, Rq) e 7^2 : 3Px,Y.u e De s.t. 
S'e = { R > I{X-U), (211) 

R + Ro > IiX,Y;U). 



using the same definition for as in ( 11801 ). Notice that 

f]Se C Closure [ Q . (212) 



£>0 \e>0 



This can be verified by contradiction. Suppose (a, b) is in the left-hand side but not the right-hand side. Find the smallest b* 
such that (a, b*) is in the right-hand side. Then b* > b. Choose e small enough to exclude (a, (5* + b)/2) from the right-hand 
side and so that g{e) < {b* — b)/2. Thus, a contradiction. 
Now define the function / : n^^\\y\\^\ as follows: 

nPx.Y,u) = {I{X-U)J{X,Y-U)). (213) 

The images f{D) and f{De) characterize the rate regions S and S'^. That is, the Pareto optimal points in the images and the 
respective rate regions are the same. Had the rate regions S and S'^ been defined with equality for the rate constraints rather 
than inequality, then they would precisely equal the images f{D) and f{De). 
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Notice that 

f]f{D,) = f{D), (214) 
oo 

because ne>o ~ ^^'■^ decreasing subsets (as e decreases) of the compact probability simplex (due to the 

cardinality bound), and / is a continuous function. This implies, 

f]S', = S. (215) 
Finally, S is closed due to / continuous and D compact. ■ 

VII. Summary 

The distributed channel synthesis problem demands unconventional analysis tools and codec constructions and yet lends 
itself to a complete information theoretic description of the achievable rate region, found in Theorem 111.11 This region reveals 
that common randomness, independent of the channel input, can replace some of the required communication rate for channel 
synthesis, reducing the communication rate from Wyner's common information C{X;Y) to Shannon's mutual information 
I{X; Y). Also, §III-EI highlights that distributed channel synthesis is as efficient as local channel synthesis in terms of random 
bits needed by the system. 

The results presented herein assume that distributed channel synthesis is designed for a specific i.i.d. input distribution. An 
obvious generalization of these results begs to be addressed. What can be done universally, for any input? It is not difficult 
to show that the intersection of the rate region iS in dHJ over all input distributions is necessary and sufficient for synthesizing 
a memoryless channel with an arbitrary input distribution (not necessarily i.i.d., but known at the time of encoder design). 
However, truly universal distributed channel synthesis should produce a high-fidelity channel output for any arbitrary input, 
not known during the design of the encoder Unfortunately, achievability for efficient universal distributed channel synthesis, 
with limited common randomness, does not follow directly from our results. 

A library of tools referred to generally as the cloud mixing lemma are found in ijlVI (some of which do apply universally, 
as stated). The basic cloud mixing lemma forms the backbone of our achievability proof, and the section also provides 
generalizations, precise bounds, and a variety of related corollaries, including the randomness requirements for local channel 
synthesis. 

The consequences of distributed channel synthesis are of interest for their application to secrecy, game theory, quantum 
measurements, and the reverse Shannon theorem. However, the proof and coding techniques are also insightful on their own. 
In particular, the achievability proof embarks on a construction of a feasible joint distribution over all parts of the system, 
without first specifying the encoder and decoder behavior. This approach is valid in general for problems in information theory. 



Appendix 

Proof of Lemma 17/7. 7 1 in miI-B\ The proof that Q D Convex Hull(f/o) follows naturally from Theorem III. 1 1 Notice that 
any point in Qq can be achieved by Player 1 first generating X" and then using the communication to synthesize a channel 
with output F". The inequalities in (O are both satisfied with {R,Ro) = {C{X;Y),0). The property of total variation stated 
in (|29] l allows us to analyze the payoff as if the actions produced are exactly i.i.d. Then time sharing gives us the convex hull 
of Go- 

For the converse statement, Q D Convex Hull(C?o), we need to rule out the possibility that some other use of the communi- 
cation, not resulting in nearly i.i.d. actions, is more beneficial. Let U represent the message used for communication. Notice 
the following: 

nR > H{U) (216) 
> (217) 



Y,I{U;XuYt\X'-\Y'-^) (218) 



t=i 



nI{U-XT,YT\X'~\Y'-',T), (219) 
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and 



n < 



1 " 

-En. 

t=i 
1 " 

-E 



min E7r(Xt,yt,z(X*-\r*-\t)) 



mm 

2 : x*-^-xy*-^Y.r^z n 



n 

-Y,¥.n{XuYuz{X'-\Y'-\t)) 

rt ^ — ^ 



t=l 



min E7:(Xt,Yt,z(X'-\Y'-'-,T)), 



(220) 

(221) 

(222) 
(223) 



where T is an independent random variable uniformly distributed on the set {l,...,n}. For simplicity, let us summarize by 
making the substitution W = T), X = Xt, and Y = Yt- Notice that we have the Markov chain X-{U, W)-Y 

by the constraints of the communication. Then, 



R > I{X,Y;U\W), 

n < min En{X,Y,z{W)). 



(224) 
(225) 



This is equivalent to the convexification of the points in Qo- ■ 
Proof of miI-C\ Points in Spc can be achieved the same way as points in S are achieved for the main result of 
Theorem III. II with the additional step of applying a one-time-pad to the communication message. 

To prove that this is optimal (converse), we first use the triangle inequality and Lemma IV. II to note that 



TV 



< 



< 2 



Px^,y-,j-PjI[QxQy\x 
^\\Px^.y^Pj-PjllQxQY\x 
Px-,y-,j-PjY[QxQy\x 
^jPx^.Yr^ ^YIQxQyix 
Px^^^y^^^j-PjYIQxQyix 



TV 



TV 
TV 



TV 



TV 



(226) 
(227) 
(228) 
(229) 
(230) 



By the definition of achievability, the right-hand side can be made arbitrarily small. We follow the steps of HVll Notice first 
that we can use Theorem 17.3.3 of 127] to bound the mutual information. 



I{X'',Y'';J) = H[X'\Y' 



H{J)- H{X'^,Y'^,J) 



where g is defined in ( II8II 1 and e is the arbitrarily small total variation tolerance of the synthesis objective. 
Now we replace the steps of ( 11931 ) with 

nRo > Hp{K) 

> Hp{K\J) 

> /p(X",r";if|J) 

> /p(X",r";J,if)-ig(2e) 



The proof is completed by following the remaining steps of i )Vll and altering the definitions of and in ( |179l l and (II8OI 1 
appropriately. ■ 
Proof of mil-Di We use the same construction as in ijV] Notice that (11411) still holds with the rates provided, and we 
adjust ( 1139b to state 



(231) 
(232) 



(233) 
(234) 
(235) 

(236) 
(237) 



lim E 

n— >-CX3 



UQxQ 



Y\X 



= Vt. 



TV 



(238) 



Furthermore, we call on the stronger conclusion from Lemma II V. 1 1 that states that the limits converge exponentially fast. 
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Taking steps analogous to ( |148l l. 



Pvt yt 



llQxQ 



Y\X 



TV 



< 



(a) 



Tjv-t yt 



Wqxq 

<:",Y",j,k\ 

YIqxq 



TV 

Y\X 



Y\X 



\P: 



X^.K — ^X".K 



ITV 



+ 



X yt v* 



l[QxQ 



TV 

Y\X 



TV 



TV 



TV 



TV 



where (a) is a consequence of Lemma IV. II and (b) uses Lemma IV. 2 1 

The expected value of the right-hand side above goes to zero exponentially fast. Therefore, 



^ \Pxt_^^Y}_^-X{QxQY\X 

t=B+l 



TV 



(n-B) 
0. 



Pyt 



iXl^Y\X 



TV 



(239) 
(240) 

(241) 
(242) 

(243) 
(244) 

(245) 

(246) 



(247) 
(248) 



Proof of miI-E\ Achievability is straightforward. The construction in SjV]is valid. We only need to locally synthesize the 
channel Py\u- This can be done with a rate Rl > H{Y\U) (see Figure |29]of illV-Db . 
For the converse we simply note that 



Rq + Rl > H(Y\X), 



(249) 



which is the required rate of randomness needed for local channel synthesis and can be verified with lIZTl Theorem 17.3.3] 
and Jensen's inequality. Finally, by the nature of Slr in ( |40| |. a constant bound on the sum rate Rq + Rl is equivalent to the 
corresponding bound on R^ because the corner point claims the entire region. That is, U can always be chosen to increase 
the requirement on Rq and decrease the requirement on R^, without costing a higher communication requirement R. ■ 
Proof of Lemma \iVT2\ in gED} 
This proof is nearly identical to the proof of Lemma IIV.2I The main difference is the definition of the typical set A'^. in 
(USB. 



w 



/(•) 


u 


^V\UM 

) 




V 









Fig. 32: Repeat of Figure \25\ Lemma II V. 121 is a generalization of Lemma IIV.2I Here a random variable W in the input to 
a deterministic encoder which produces an output U . A channel then acts on the pair {W, U). Given any output distribution 
consistent with the source and channel, Lemma llV. 12l bounds the expected total variation between the desired output distribution 
and the distribution induced by a randomly constructed encoder. 



Recall that we are given three distributions: the source distribution $vK> the codebook distribution $t/|vi/, and the channel 
^v\w,u- The source and channel are stochastic according to their prescribed distributions. However, the encoder produces a 
deterministic output u{w). The induced output distribution is 

Pv{v) = ^w{w)^v\w,u{v\w,u{w)). (250) 

tuew 

The lemma bounds the total variation between the desired output distribution Qy — $y that would result from a stochastic 
encoder that operates according to ^u\w and the output distribution induced by the deterministic encoder in ( |250t . Specifically, 
we bound the expected total variation when the codebook entries are generated independently according to the desired 
conditional distribution ^u\w 
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The proof is stated in terms of discrete random variables, but each step holds for continuous random variables as well by 
replacing the appropriate sums with integrals. 

We start by categorizing {w,u,v) triples into a typical set that will behave well under using the information density 
U ; V) and self-information i^{W) = iq,{W; W) as the criterion: 

A'^ ^ {{w,u,v) : - < r} . (251) 

The idea will be to separate out the contribution to Py coming from typical pairs {w, u, v) £ A'j. from the rest. Let us 
define two functions on V that sum to Py'. 

\w.u{v\w,U{w)) l{{w,U{w),v) e A'^) , (252) 

wew 

\w,uiv\u!,U{w)) 1 {{w,U{w),v) ^ A'^) , (253) 

wew 

where 1 represents the indicator function, and U{w) is the codebook entry for source w and is a capital letter to represent that 
the codebook is randomly generated. Under these definitions, 

Pv = P1+P2. (254) 

An important observation about the induced output distribution Py, which is random because the codebook is random, is 
that it is unbiased with respect to the desired output distribution Qv- 



(a) 

wew 

(b) 



EPv{v) = ^ $w(w)E$yn4/,c;(v|u;,{/(m)) (255) 
wew 

Y ^w{w)Y'^u\w{u\w)<^>v\w,u{v\w,u) (256) 
wew ueu 

= (257) 

= Qv{v). (258) 

where (a) is an application of linearity of expectation to the definition of Py, and (b) is due to the independence of the 
codebook. 

We separate the total variation E ||Py — QyHj-y into two parts: 

^\\Pv-Qv\\tv = E||Pv'-EFy||^^ 

= i^E|Pv(«)-EPy(«)| 

vev 

< i^E|PiH-EPi(t;)| 

vev 



i^E|P2H-EP2(^;)|, (259) 



2 

vev 



where (a) is due to the triangle inequality. 

The first sum in (|259t is the interesting one to consider, so we save it for last. The second sum is easy to handle and is 
small as long as the typical set A'^ is likely. Starting by again making use of the triangle inequality, 



^Y^\P2iv)-EP2iv)\ < Y^P2iv) 

vev vev 



(260) 



^ E ^ <^>w{w)<Pv\w,u{v\w, U{m)) 1 ((w, U{w), v) i A!,) (261) 
vev \wew / 

Y E *wME {^v\w,uiv\w, U{m)) 1 ((w, f/H, v) i M,)) (262) 
vev wew 

Y I] ^w(w) Y ^v\w{u\w)^y\w,uHw, u) 1 ((w, u, v) i A'^) (263) 

vev wew ueu 

1-P*(^;) (264) 

P* {(.i^w,u,v {W, U- V) - [W) > t) . (265) 



38 



The remaining term in (|259l l deals with only typical pairs. To bound this term we appeal to a variance bound with the help 
of Jensen's inequality: 



E|Pi(«)-EPi(i;)| < VE(Pi(«) -EPi(t.))2 (266) 



= VVar Pi(v). (267) 
Since the input codebook is a collection of inputs generated i.i.d., the variance of Pi{v) breaks down nicely as 

Var = Var I ^ <S>wH^v\w,uiv\w,U{m)) l{{w,U{w),v) £ j (268) 

*V(^«)Var {'^>v\w,u{v\w, U{w)) 1 {{w, U{w), v) G A,)) (269) 

wew 

< ^^w{w) E {<fv\w,u{v\w, U{w)) 1 {{w, Uiw), v) e A',))^ (270) 

luew 

= J2 ^^wiw)J2^u\wiu\w){^v\w,uiv\w,u)liiw,u,v) eA'^)y (271) 

luew ueu 

^ Y '^V(w) Y '^u\w{u\w)^^v\w,uiv\w,u) (272) 

= Y ^w{w)^w,u,v{w,u,v)^v\w,u{v\w,u) (273) 

{w.u) : [w ,u,v)^A'^ 

< Y 2^<i>w,u{w,u)<i>v{v)<^>v\w,u{v\w,u) (274) 

(w.u) : (w ,u.v)^A.i^ 

= 2^$y(v) Y ^w,u.viw,u,v) (275) 

< 2^$V(w)- (276) 

Equality (a) is due to the independence of the items in the codebook, and (b) comes from the definition of A'^. 
The conclusion with respect to the first term in (1259b is 



^-Y^\P,{v)-EP,{vn\ < i5^y2^¥VR (277) 

uGV vev 

^2^/^ (278) 
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